Some digging in the code and pulling hairs out lead to the issue: include_once was failing us. Yes, it was doing include of the same file again! One little detail – that only happened when we used both relative and absolute include path, something like:

include_once("../includes/fancyLibrary.php");
and
include_once("/absolute/path/to/document/root/includes/fancyLibrary.php");

Of course that should not be an issue, according to the PHP documentation.
When I started troubleshooting those includes calling
realpath("../includes/fancyLibrary.php");
the truth popped out – realpath was not returning anything. And then we found why – there was missing “read” attribute on the document root directory, it only had executable bit set (711). Changing permissions to 755 (added read attribute’s bit) fixed the issue.

So, problems with realpath or include_once PHP functions? Check folders permissions!

PHP script which was doing ftp_connect was failing on some installations with Segmentation Fault (core dumped) message. After quite a bit of time wasted looking for answers why, I found that problem was with the FTP host name having trailing line break. Well, that host name was coming from config and technically only developers had access to it, so looks like we assumed that variable “safe” and were not doing standard checks on special characters. To make things worse that line break is hard to see with debug output, because that line break is too easy to miss when you check variable value in debugger.

I am still not sure why Segmentation Fault (instead of proper message) – must be some PHP bug (and that server has Solaris OS).

Anyway, hope that helps somebody

Error 400
Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

That means that your browser is trying to send too much at once. Probably, it’s size of the cookies. The easy way to fix it – delete cookies. Of course, you lose all info in the cookies, but at least you can access site again.

If you develop/manage web site and your customers complain of this error message – there are two things to do:

• Review cookies and limit the size of cookies your application stores in the browser.
• Increase allowed request size in web server configuration

To increase request size in Apache, add directive LimitRequestFieldSize to the configuration file. Default value is 8190 bytes. To increase it 2 times add
LimitRequestFieldSize 16380
Apache restart is required.

I am able to connect from local shell, so first thought was if something wrong with recent Zend Framework upgrade, but after a while I figured out that answer is very simple – SELinux was blocking remote connections from PHP scripts executed by Apache web server. The error code (13) at the end of error message means “permission denied”, so that’s the indication to see if you have similar issue or not.

Anyway, login as root and do
setsebool -P httpd_can_network_connect=1
to make it work.

Of course, think twice because you make web server a bit less secure, so don’t do that unless you are sure you need it.