302 redirect to Main/WebHome after XHR to connector

Dear Foswiki, I am using TimLegge's docker-foswiki in a katacoda environment.

When using the Wiki workbench I get the following message when trying to display the component grid.

DataTables warning: table id=DataTables_Table_0 - Invalid JSON response. For more information about this error, please see http://datatables.net/tn/1

The url suggest a debugging technique which gives the following results (Firefox 74 - 64bit, Windows 7):

In the Console tab this shows:

Loading mixed (insecure) active content “http://2886795279-80-cykoria04.environments.katacoda.com/bin/rest/JQDataTablesPlugin/connector” on a secure page
Loading mixed (insecure) active content “http://2886795279-80-cykoria04.environments.katacoda.com/pub/System/JQDataTablesPlugin/i18n/en.js” on a secure page

In the Network tab the actions for refresh page are listed as follows:

200 GET   2886795279-80-cykoria04.environments.katacoda.com   DataForm   document   html   48.28 KB   47.54 KB
304 GET 2886795279-80-cykoria04.environments.katacoda.com   7101620184dac80ad88e0a31460489c6.css   stylesheet   css   cached   399.82 KB
--- many more 304 ---
302 POST 2886795279-80-cykoria04.environments.katacoda.com   connector   xhr   html   26.26 KB   25.89 KB
--- many more 304 ---
   xhr   js   cached   277 B
304 GET   2886795279-80-cykoria04.environments.katacoda.com   en.js   xhr   js   cached   862 B
200 GET   2886795279-80-cykoria04.environments.katacoda.com   WebHome

The 302 is a post asking for the JQDataTablesPlugin/connector

https://2886795279-80-cykoria04.environments.katacoda.com/bin/rest/JQDataTablesPlugin/connector

It is interesting, because it returns HTML!! (according to the Type collumn on the debugger Network tab)

Request headers:

Host: 2886795279-80-cykoria04.environments.katacoda.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 1598
Origin: https://2886795279-80-cykoria04.environments.katacoda.com
DNT: 1
Connection: keep-alive
Cache-Control: max-age=0
TE: Trailers

Request Body:

draw=1&columns%5B0%5D%5Bdata%5D=Topic&columns%5B0%5D%5Bname%5D=Topic&columns%5B0%5D%5Bsearchable%5D=true&columns%5B0%5D%5Borderable%5D=true&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B1%5D%5Bdata%5D=index&columns%5B1%5D%5Bname%5D=index&columns%5B1%5D%5Bsearchable%5D=false&columns%5B1%5D%5Borderable%5D=false&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B2%5D%5Bdata%5D=TopicTitle&columns%5B2%5D%5Bname%5D=TopicTitle&columns%5B2%5D%5Bsearchable%5D=true&columns%5B2%5D%5Borderable%5D=true&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B3%5D%5Bdata%5D=Summary&columns%5B3%5D%5Bname%5D=Summary&columns%5B3%5D%5Bsearchable%5D=true&columns%5B3%5D%5Borderable%5D=true&columns%5B3%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B3%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B4%5D%5Bdata%5D=WikiApplication&columns%5B4%5D%5Bname%5D=WikiApplication&columns%5B4%5D%5Bsearchable%5D=true&columns%5B4%5D%5Borderable%5D=true&columns%5B4%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B4%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B5%5D%5Bdata%5D=Changed&columns%5B5%5D%5Bname%5D=Changed&columns%5B5%5D%5Bsearchable%5D=true&columns%5B5%5D%5Borderable%5D=true&columns%5B5%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B5%5D%5Bsearch%5D%5Bregex%5D=false&order%5B0%5D%5Bcolumn%5D=5&order%5B0%5D%5Bdir%5D=desc&start=0&length=10&search%5Bvalue%5D=&search%5Bregex%5D=false&connector=search&webs=Applications%2FTestApp&form=&topic=Applications%2FTestApp.DataForm&t=1585372959&query=TopicType%3D~'%5CbDataForm%5Cb'+

Request response:

HTTP 302 Found
server: nginx/1.15.0
date: Sat, 28 Mar 2020 05:22:43 GMT
content-type: text/html; charset=utf-8
content-length: 0
access-control-allow-origin: *
location: http://2886795279-80-cykoria04.environments.katacoda.com/Main/WebHome
set-cookie: FOSWIKISID=8dc5a0355b3febf3897755264227c714; path=/; HttpOnly
via: 1.1 google
alt-svc: clear
X-Firefox-Spdy: h2

I think the issue is that the response redirects to Main/WebHome. But WHY??? And what can I do about it?

Thanks in advance for your advice

-- BramVanOosterhout - 28 Mar 2020

Update 29 March

I did a comaprison between a successful request on my home server and the failed request on the katacoda environment/server.

The difference is that the katacoda server lacks

X-Requested-With: XMLHttpReques

in the header.

Stack overflow tells me that
"A good reason is for security - this can prevent CSRF attacks because this header cannot be added to the AJAX request cross domain without the consent of the server via CORS
https://stackoverflow.com/questions/17478731/whats-the-point-of-the-x-requested-with-header

So the root cause is probably myserver config related to Cross-Origin Resource Sharing. I'll keep you posted.

-- BramVanOosterhout - 29 Mar 2020

Update 30 March

Well, I do not get any bad messages any more. I dealt with CORS as recommended. I used brute force and will refine later. Here is what I added to the nginx server configuration:

add_header Access-Control-Allow-Origin  *;
add_header Access-Control-Request-Methods: "PUT,GET,POST,OPTIONS";
add_header Access-Control-Allow-Credentials: true;
add_header Access-Control-Allow-Headers "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range";
add_header Access-Control-Expose-Headers "Content-Length,Content-Range";

I still don't get any content returned in the call to the connector. But no ugly messages in the Firefox debugger. And no ugly messages in the error log (I think).

The error log has a notice:

2020/03/30 05:16:47 [notice] 106#106: *54 "^SiteSucker|^iGetter|^larbin|^LeechGet|^RealDownload|^Teleport|^Webwhacker|^WebDevil|^Webzip|^Attache|^SiteSnagger|^WX_mail|^EmailCollector|^WhoWhere|^Roverbot|^ActiveAgent|^EmailSiphon|^CrownPeak-HttpAgent|^$" does not match "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0", client: 172.17.0.5, server: _, request: "POST /bin/rest/JQDataTablesPlugin/connector HTTP/1.1", host: "2886795331-80-elsy05.environments.katacoda.com"

Should I be worried about this?

-- BramVanOosterhout - 30 Mar 2020

Update 1 April

No joke!

I cried victory too early. Sure, the browser message went away.So I can operate parts of the wiki workbench. But the data table does not work. And the browser console still shows the earlier problems:

Still no X-Requested-With: XMLHttpRequest. Still a redirect to Main.webHome. And that redirect now fails with a CORS message.

From all the things I have read I expect an OPTIONS request in the browser log. But that does not appear. Anyone knows whether that will appear in the browser log? It does not show in the web server log either. But if it does not get sent, why not?

Anyway, I am now down to the debugger. I'll first try to understand the flow on my home server and then back to Katacoda.

Thanks for listening so far. Any suggestions are welcome.

-- BramVanOosterhout - 01 Apr 2020

Might be related to the mixed http -> https situation that you seem to be in... Have you got SecurityHeadersPlugin installed?

-- MichaelDaum - 14 Apr 2020

Hi Michael, Yes, SecurityHeadersPlugin is installed in Tim's docker-foswiki.

I turned it off and get the same result.

• The connector does not return data; and
• And the Firefox console shows: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://2886795335-80-cykoria03.environments.katacoda.com/Main/WebHome. (Reason: CORS request did not succeed).

I had issues with the http/https transition before (Support.Question2025). But that was resolved with: {Sessions}{UseIPMatching} off

-- BramVanOosterhout - 14 Apr 2020

Hi

You are on the right track with the headers I believe. I have a ngnxl_ssl -> docker_foswiki-http->fastcgi so it might be similar. I believe I have to addThe X-Forwarded-For or something like that. I will try to look in the morning

-- TimothyLegge - 15 Apr 2020

In my install with a nginx proxing to docker-foswiki I needed the following added to my nginx proxy

    proxy_pass              http://docker-foswiki;
    proxy_http_version      1.1;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Scheme https;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Server $host;
    proxy_read_timeout     90;

-- TimothyLegge - 16 Apr 2020

Thanks Tim,

I hope to be able to try this this weekend.

-- BramVanOosterhout - 17 Apr 2020

Hi Tim, Michael.

This issue has been resolved.

Tim, thanks for your encouragement and the docker-foswiki image. In the end the issue was related to the foswiki configuration.

Michael, you were right. The https->http transition was the root cause of the Cross Origin Request Sharing issue. I needed to configure Foswiki so that Foswiki made sure that all browser requests (inc Ajax request) were identical. Since Katacoda makes the requests using https, Foswiki needs to tell JQDataTables to issue all https requests, event though the requests are received as http on port 80.

Here my my complete docker image setup:

First: Run a script to install the image. When that is done, report completion by creating the Foswiki_config_complete file.

#!/bin/bash
if [[ $(docker inspect foswiki 2>/dev/nul) == '[]' ]]; then
docker run -d --name foswiki -p 80:80 timlegge/docker-foswiki
fi
until [ "`/usr/bin/docker inspect -f {{.State.Running}} foswiki`"=="true" ]; do sleep 1; done
touch Foswiki_config_complete 

Second: Run a script to update the Foswiki configuration in bash inside the foswiki container:

/dev/null
echo " >>"
##
## Provide acces for the admin user
##
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "touch data/.htpasswd"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {Password}='password'"
##
## Set the configuration to force all requests generated by Foswiki to be https
##   {ForceDefaultUrlHost}='1'
##   {DefaultUrlHost}='https://$1'  -- $1 is the katacoda provided external url for the container
##
## And accept request in http
##   {PermittedRedirectHostUrls}='http://$1,http://localhost'
##
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {Sessions}{UseIPMatching}='0'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {DefaultUrlHost}='https://$1'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {ForceDefaultUrlHost}='1'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {PermittedRedirectHostUrls}='http://$1,http://localhost'"
##
## Fix some configuration settings
##
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {JQDataTablesPlugin}{DefaultConnector}='dbcache'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {Plugins}{AutoViewTemplatePlugin}{Enabled}='0'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "tools/configure -save -set {Plugins}{SolrPlugin}{Enabled}='0'"
docker exec -it -w /var/www/foswiki foswiki /bin/bash -c "sed -i 's/Set SKIN/#Set SKIN/' data/Main/SitePreferences.txt"
##
## Update the foswiki cache
##
docker exec -it -w /var/www/foswiki/bin foswiki /bin/bash -c "./view /Main/Webhome -refresh=all"
##
## And allow the Cross Origin Resource Sharing (https -> http) 
docker exec -it foswiki /bin/bash -c "sed -i '/server_name/a      add_header \"Access-Control-Allow-Origin\"  *;' /etc/nginx/conf.d/default.conf"
docker exec -it foswiki /bin/bash -c "/usr/sbin/nginx -s reload"
##
## Show a bash shell in the foswiki container
##
docker exec -it -w /var/www/foswiki foswiki /bin/bash
echo " done."
##
## And we're done!!
##

Thanks all, for listening, the hints and the support.

-- BramVanOosterhout - 24 Apr 2020

Commenting is disabled while not logged in