Page 1 of 1

nfsv4 expired locks lost

Posted: 03 Nov 2015 03:41
by tuaris
I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open
The server does get busy, but not overloaded:
Image

There are about 4 FreeBSD 9.3 boxes that connect to this NFSv4 server and the IO almost always at least 100 mbps (with some peaks to 600+mbits) 24/7.

The system is (I think) well capable of handling the load:
Image

Image

Re: nfsv4 expired locks lost

Posted: 03 Nov 2015 11:48
by daoyama
tuaris wrote:I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open
I didn't see that message, but there is some tunable parameters.

The default value is:

Code: Select all

vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
Try to increase from loader.conf.
I'm interesting about vfs.nfsd.v4statelimit.
Verify it by CLI:
# sysctl vfs.nfsd

Re: nfsv4 expired locks lost

Posted: 03 Nov 2015 21:48
by tuaris
The current settings are:

Code: Select all

sysctl vfs.nfsd
vfs.nfsd.request_space_throttle_count: 0
vfs.nfsd.request_space_throttled: 0
vfs.nfsd.request_space_low: 31457280
vfs.nfsd.request_space_high: 47185920
vfs.nfsd.request_space_used_highest: 1657432
vfs.nfsd.request_space_used: 4344
vfs.nfsd.groups: 1
vfs.nfsd.threads: 32
vfs.nfsd.maxthreads: 32
vfs.nfsd.minthreads: 32
vfs.nfsd.fha.fhe_stats: No file handle entries.
vfs.nfsd.fha.max_reqs_per_nfsd: 0
vfs.nfsd.fha.max_nfsds_per_fh: 8
vfs.nfsd.fha.bin_shift: 22
vfs.nfsd.fha.enable: 1
vfs.nfsd.cachetcp: 1
vfs.nfsd.tcpcachetimeo: 43200
vfs.nfsd.udphighwater: 500
vfs.nfsd.tcphighwater: 0
vfs.nfsd.enable_stringtouid: 0
vfs.nfsd.debuglevel: 0
vfs.nfsd.enable_locallocks: 0
vfs.nfsd.issue_delegations: 0
vfs.nfsd.commit_miss: 0
vfs.nfsd.commit_blks: 0
vfs.nfsd.mirrormnt: 1
vfs.nfsd.async: 0
vfs.nfsd.server_max_nfsvers: 4
vfs.nfsd.server_min_nfsvers: 2
vfs.nfsd.nfs_privport: 0
vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
vfs.nfsd.enable_nogroupcheck: 1
vfs.nfsd.enable_nobodycheck: 1
vfs.nfsd.disable_checkutf8: 0
Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000

Re: nfsv4 expired locks lost

Posted: 04 Nov 2015 01:49
by daoyama
tuaris wrote: Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000
I don't know. At least, hash size must set before loading kernel.

Re: nfsv4 expired locks lost

Posted: 04 Nov 2015 02:48
by tuaris
What value do you recommend I set for the hash size and state limit?
Is their an NFSv4 tuning guide I can read over?

Re: nfsv4 expired locks lost

Posted: 05 Nov 2015 00:08
by Parkcomm
Dayoma - could you explain what is causing the problem the changing the vfs.nfsd.v4statelimit solves and how it might come about?

(FYI - I've seen this problem before and ended up "solving" it by falling back to NFSV3 - not ideal)

Re: nfsv4 expired locks lost

Posted: 21 Apr 2018 08:30
by tuaris
I've experienced some data loss as a result of this problem. An application running on one of the NFS clients continued to write to a file and the data was never 'saved'.

I've switched back to NFSv3. Looks like NFSv4 isn't ready.

According to this page, this appears to be what's happening.
https://access.redhat.com/solutions/1179643