"Synchronizer error", Clank – best practices?

I installed Oro CRM on a digital ocean droplet, and am running it in production mode. For the most part the process was nice and smooth, but I have come across two stumbling blocks which are not addressed in the documentation, and I’m not sure how to resolve them.

There is a “Synchronizer can’t connect to server.” error displayed when anyone logs in. I don’t see any corresponding errors in the apache2 or /app/logs/prod.log files, and I’m not sure what this error means.

The second (possibly related) is that the suggested command to run the Clank server:

php /var/www/oro-crm/app/console clank:server --env prod

1	php /var/www/oro-crm/app/console clank:server --env prod

Seems inadequate; the process stays attached to the terminal, has debug-ish output, and no process monitoring. To run it in production I have simply “nohup’d” it, and redirected input from /dev/null – but perhaps I am missing something? Is there a suggested php process monitor or other app/console command that is more production appropriate?

Is there an upstart profile, or popular PHP process monitor I could adapt for it? I’m more used to the Ruby ecosystem where this kind of thing would be done with bluepill or god.

Topic

[Yevhen ShyshkinParticipant]

Hello, Aaron.

For now Clank server should be started manually (as you said). I’m using app/console clank:server > /dev/null 2>&1 & to silence all output. We need to improve this behaviour and provide easier way to work with it.

In general, if you want to daemonize clank server process, you should work with this command as with regular shell command and use any external tool or shell script to run it as a daemon (f.e. http://software.clapper.org/daemonize/).

[AaronParticipant]

Thanks, Yevhen, although that still leaves me wondering about the “Synchronizer can’t connect to server.” error when any user logs in.

In case other people stumble across this discussion before changes are rolled into the console command, here is the solution I’m using on Ubuntu/Linux:

Create a shell file to launch the console command with nohup, and use absolute paths. Write the nohup output to get a pidfile.

Then you can use monit (example monit config) to watch the pidfile and get basic process management.

This should work on any *NIX system which monit works on.

[Yevhen ShyshkinParticipant]

“Synchronizer can’t connect to server.” error might appear if server port (default is 8080) is closed or filtered by firewall. Please, ensure that port is really opened f.e. using nmap:

user@host:~$ nmap -sT -p 8080 127.0.0.1 Starting Nmap 5.21 ( http://nmap.org ) at 2014-07-02 12:31 GET Nmap scan report for localhost (127.0.0.1) Host is up (0.000038s latency). PORT STATE SERVICE 8080/tcp open http-proxy Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

1 2 3 4 5 6 7 8 9 10

user@host:~$ nmap -sT -p 8080 127.0.0.1   Starting Nmap 5.21 ( http://nmap.org ) at 2014-07-02 12:31 GET Nmap scan report for localhost (127.0.0.1) Host is up (0.000038s latency). PORT     STATE SERVICE 8080/tcp open  http-proxy   Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

[AaronParticipant]

hmm, no, nmap reports the port is open:

root@crm:~# nmap -sT -p 8080 127.0.0.1 Starting Nmap 6.40 ( http://nmap.org ) at 2014-07-02 13:27 EDT Nmap scan report for localhost (127.0.0.1) Host is up (0.00036s latency). PORT STATE SERVICE 8080/tcp open http-proxy Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

1 2 3 4 5 6 7 8 9

root@crm:~# nmap -sT -p 8080 127.0.0.1   Starting Nmap 6.40 ( http://nmap.org ) at 2014-07-02 13:27 EDT Nmap scan report for localhost (127.0.0.1) Host is up (0.00036s latency). PORT     STATE SERVICE 8080/tcp open  http-proxy   Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

Also clank appears to be running fine:

root@crm:~# ps -auxwww | grep clank root 5289 0.0 2.9 348512 29564 ? S Jun26 0:06 /usr/bin/php /var/www/oro-crm/app/console clank:server --env prod root 31806 0.0 0.0 11748 924 pts/0 S+ 13:28 0:00 grep --color=auto clank

1 2 3

root@crm:~# ps -auxwww | grep clank root      5289  0.0  2.9 348512 29564 ?        S    Jun26   0:06 /usr/bin/php /var/www/oro-crm/app/console clank:server --env prod root     31806  0.0  0.0  11748   924 pts/0    S+   13:28   0:00 grep --color=auto clank

And nothing in prod.log or syslog relating to clank.

[Yevhen ShyshkinParticipant]

Ok, let’s check everything from the very beginning.

Clank server parameters at app/config/parameters.yml:

websocket_host: 127.0.0.1 websocket_port: 8080

1 2	websocket_host: 127.0.0.1 websocket_port: 8080

Please, ensure that client part of the application (browser) aware about websocket_host (i.e. this host must be resolved by client to server’s IP) and port is open for client.

Init connection headers (can be found using firebug):

Request Headers GET / HTTP/1.1 Host: 127.0.0.1:8080 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Sec-WebSocket-Version: 13 Origin: http://crm.dev Sec-WebSocket-Protocol: wamp Sec-WebSocket-Key: S1GdjGXRYkx1VrXSEcE59Q== Connection: keep-alive, Upgrade Pragma: no-cache Cache-Control: no-cache Upgrade: websocket Response Headers HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade sec-websocket-accept: SCYXMwMuvCzf2j5q3ZqGNT4xPF8= Sec-WebSocket-Protocol: wamp X-Powered-By: Ratchet/0.2.7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Request Headers GET / HTTP/1.1 Host: 127.0.0.1:8080 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Sec-WebSocket-Version: 13 Origin: http://crm.dev Sec-WebSocket-Protocol: wamp Sec-WebSocket-Key: S1GdjGXRYkx1VrXSEcE59Q== Connection: keep-alive, Upgrade Pragma: no-cache Cache-Control: no-cache Upgrade: websocket   Response Headers HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade sec-websocket-accept: SCYXMwMuvCzf2j5q3ZqGNT4xPF8= Sec-WebSocket-Protocol: wamp X-Powered-By: Ratchet/0.2.7

Also you can trace request and response with any HTTP sniffer (f.e. Wireshark).

Clank server output:

Starting Clank Launching Ratchet WS Server on: 127.0.0.1:8080 1282 connected 1282 disconnected 1344 connected 1344 disconnected ...

1 2 3 4 5 6 7 8

Starting Clank Launching Ratchet WS Server on: 127.0.0.1:8080 1282 connected 1282 disconnected 1344 connected 1344 disconnected ...

If something not works – please, put here appropriate logs.

[AaronParticipant]

Thanks for the details.

The problem seems to be that on the browser a request to load a bundle is failing:

Unable to load external script. http://crm.baymaterials.com/ Line 3702

1 2 3 4 5

Unable to load external script. http://crm.baymaterials.com/ Line 3702

In particular it is trying to load
/bundle/orocrmrequest/js/embed.form.js


Interestingly my install has no “orocrmrequest” directory in the bundles, so that explains the failure. . .I don’t know why my install doesn’t have it, however.

Since I don’t know what is in that file, it may or may not cascade into the later error:

ab._construct = function (url, protocols) { if ("WebSocket" in window) { // Chrome, MSIE, newer Firefox if (protocols) { return new WebSocket(url, protocols); } else { return new WebSocket(url); } } else if ("MozWebSocket" in window) { // older versions of Firefox prefix the WebSocket object if (protocols) { return new MozWebSocket(url, protocols); } else { return new MozWebSocket(url); } } else { return null; } };

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

ab._construct = function (url, protocols) {    if ("WebSocket" in window) {       // Chrome, MSIE, newer Firefox       if (protocols) {          return new WebSocket(url, protocols);       } else {          return new WebSocket(url);       }    } else if ("MozWebSocket" in window) {       // older versions of Firefox prefix the WebSocket object       if (protocols) {          return new MozWebSocket(url, protocols);       } else {          return new MozWebSocket(url);       }    } else {       return null;    } };

According to firebug the ‘url’ variable isn’t defined when that gets called, which leads the client to attempt to open a websocket to a default of

I had the breakpoint set wrong. ‘url’ is not undefined, it is being set to ‘127.0.0.1’ somewhere else in the code. I see a ‘127.0.0.1’ hardcoded on line 505 of the page:

sync(new Wamp({ secure: false, host: '127.0.0.1', port: 8080, maxRetries: 3, retryDelay: 30000 }));

1 2 3 4 5 6 7 8 9

sync(new Wamp({             secure: false,             host: '127.0.0.1',             port: 8080,             maxRetries: 3,             retryDelay: 30000         }));

But I don’t know if that value is getting threaded through to the websocket request. I’m not really set up to debug minified js here.

Regardless, it seems like the websocket request should open <the server>:8080 instead of 127.0.0.1:8080 Although the former may be passing tests if you are testing with the server on the local machine. . .

So, what is up with that missing bundle file – should I attempt a new install on another machine and then copy the bundle over?

[AaronParticipant]

Sooo I went out of the frying pan, into the frier.

I decided to migrate to git tag 1.2.0-rc1 to see if the bundle issue was straightened out there.

I pulled, checked out the relevant tag, updated composer deps, and then ran the console update. I got a giant fireball of fail back. I’m going to revert the VM to yesterday’s backup. . .

php app/console oro:platform:update --env=prod PHP Warning: Cannot redeclare class in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 ATTENTION: Database backup is highly recommended before executing this command. Please make sure that application cache is up-to-date before run this command. Use cache:clear if needed. To force execution run command with --force option: oro:platform:update --force root@crm:/var/www/oro-crm# php app/console oro:platform:update --force --env=prod PHP Warning: Cannot redeclare class in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 PHP Warning: Cannot redeclare class in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 Process migrations... > Oro\Bundle\EntityExtendBundle\Migration\RefreshExtendCacheMigration [Doctrine\Common\Persistence\Mapping\MappingException] Class 'OroEntityProxy\OroEmailBundle\EmailAddressProxy' does not exist oro:entity-extend:update-config [RuntimeException] The command terminated with an exit code: 1. oro:migration:load [--force] [--dry-run] [--show-queries] [--bundles[="..."]] [--exclude[="..."]] [RuntimeException] The command terminated with an exit code: 1. oro:platform:update [--force] [--timeout[="..."]]

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

php app/console oro:platform:update --env=prod PHP Warning:  Cannot redeclare class  in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 ATTENTION: Database backup is highly recommended before executing this command.            Please make sure that application cache is up-to-date before run this command.            Use cache:clear if needed.   To force execution run command with --force option:     oro:platform:update --force root@crm:/var/www/oro-crm# php app/console oro:platform:update --force --env=prod PHP Warning:  Cannot redeclare class  in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 PHP Warning:  Cannot redeclare class  in /var/www/oro-crm/vendor/oro/platform/src/Oro/Bundle/EntityExtendBundle/Tools/ExtendClassLoadingUtils.php on line 77 Process migrations...   > Oro\Bundle\EntityExtendBundle\Migration\RefreshExtendCacheMigration     [Doctrine\Common\Persistence\Mapping\MappingException]   Class 'OroEntityProxy\OroEmailBundle\EmailAddressProxy' does not exist   oro:entity-extend:update-config     [RuntimeException]   The command terminated with an exit code: 1.   oro:migration:load [--force] [--dry-run] [--show-queries] [--bundles[="..."]] [--exclude[="..."]]     [RuntimeException]   The command terminated with an exit code: 1.   oro:platform:update [--force] [--timeout[="..."]]

[Yevhen ShyshkinParticipant]

Hm… In template host has valid value:

sync(new Wamp({ secure: {{ app.request.headers.get('X-Forwarded-Proto') == 'https' ? 'true' : 'false' }}, host: '{{ ws.host == '*' ? app.request.headers.get('host') : ws.host }}', port: {{ ws.port }}, maxRetries: 3, retryDelay: 30000 }));

1 2 3 4 5 6 7

sync(new Wamp({     secure: {{ app.request.headers.get('X-Forwarded-Proto') == 'https' ? 'true' : 'false' }},     host: '{{ ws.host == '*' ? app.request.headers.get('host') : ws.host }}',     port: {{ ws.port }},     maxRetries: 3,     retryDelay: 30000 }));

Just checked on the latest master – it uses host from parameters.yml. If you changed web socket parameters – please, update your cache (app/console cache:clear –env prod).

As for update – please, remove app/cache/* and then run oro:platform:update.

[AaronParticipant]

I think it is just a matter of documentation/defaults in the config file, then.

config/parameters.yml should only have websocket_host = 127.0.0.1 if your server and client are on the same machine – otherwise that setting should be updated to reflect the server ip or dns, right? a default of #{env hostname} (or similar) might be more sensible, or explaining that value in the install documentation.

After setting the line to websocket_host = the-host-dns-name and resetting the cache the error seems to be gone – many thanks!

As for upgrading, I reset the VM to the previous image and tried again:

root@crm:/var/www/oro-crm# git pull Already up-to-date. root@crm:/var/www/oro-crm# git checkout HEAD Your branch is up-to-date with 'origin/master'. root@crm:/var/www/oro-crm# php composer.phar install --prefer-dist --no-dev Loading composer repositories with package information Installing dependencies from lock file Nothing to install or update Generating autoload files Updating the "app/config/parameters.yml" file root@crm:/var/www/oro-crm# rm -rf app/cache/* web/js/* web/css/* root@crm:/var/www/oro-crm# php app/console oro:platform:update --env=prod ATTENTION: Database backup is highly recommended before executing this command. Please make sure that application cache is up-to-date before run this command. Use cache:clear if needed. To force execution run command with --force option: oro:platform:update --force

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

root@crm:/var/www/oro-crm# git pull Already up-to-date. root@crm:/var/www/oro-crm# git checkout HEAD Your branch is up-to-date with 'origin/master'. root@crm:/var/www/oro-crm# php composer.phar install --prefer-dist --no-dev Loading composer repositories with package information Installing dependencies from lock file Nothing to install or update Generating autoload files Updating the "app/config/parameters.yml" file root@crm:/var/www/oro-crm# rm -rf app/cache/* web/js/* web/css/* root@crm:/var/www/oro-crm# php app/console oro:platform:update --env=prod ATTENTION: Database backup is highly recommended before executing this command.            Please make sure that application cache is up-to-date before run this command.            Use cache:clear if needed.   To force execution run command with --force option:     oro:platform:update --force

running php app/console oro:platform:update –force succeeded this time, perhaps a test user had hit the site between my clearing the cache and attempting the update last time, since I was lazy and hadn’t stopped the server first.

Thanks for all the help, and hopefully this thread is useful to people who run into the same error!

[Author]

Replies