Ran into a not-so-obvious issue with Elasticsearch today. After an update from 6.2.x to 6.4.x, an elasticsearch instance failed to start with following error in its logs.
$ tail -f instance.log
...
Caused by: java.security.AccessControlException:
access denied ("java.io.FilePermission" "/etc/elasticsearch/production/scripts" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
...
The logs indicated it couldn’t read from /etc/elasticsearch/production/scripts, but all permissions were OK. Apparently, Elasticsearch doesn’t follow symlinks for the scripts directory, even if it has all the required read permissions.
In this case, that scripts directory was indeed a symlink to a shared resource.
$ ls -alh /etc/elasticsearch/production/scripts /etc/elasticsearch/production/scripts -> /usr/share/elasticsearch/scripts
The fix was simple, as the shared directory didn’t contain any scripts to begin with.
$ rm -f /etc/elasticsearch/production/scripts $ mkdir /etc/elasticsearch/production/scripts $ chown elasticsearch:elasticsearch /etc/elasticsearch/production/scripts
Turn that symlink into a normal directory & restart your instance.
If you run multiple instances, you’ll have to repeat these steps for every instance that fails to start.