Sorry we lost some posts because of database errors!

*New 12.1 series Release:
2020-09-01: XigmaNAS 12.1.0.4.7728 - released

*New 11.4 series Release:
2020-08-27: XigmaNAS 11.4.0.4.7718 - released!


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

nfsv4 expired locks lost

Network filesystems.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
User avatar
tuaris
experienced User
experienced User
Posts: 85
Joined: 19 Jul 2012 21:31
Contact:
Status: Offline

nfsv4 expired locks lost

#1

Post by tuaris »

I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open
The server does get busy, but not overloaded:
Image

There are about 4 FreeBSD 9.3 boxes that connect to this NFSv4 server and the IO almost always at least 100 mbps (with some peaks to 600+mbits) 24/7.

The system is (I think) well capable of handling the load:
Image

Image

User avatar
daoyama
Developer
Developer
Posts: 422
Joined: 25 Aug 2012 09:28
Location: Japan
Status: Offline

Re: nfsv4 expired locks lost

#2

Post by daoyama »

tuaris wrote:I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open
I didn't see that message, but there is some tunable parameters.

The default value is:

Code: Select all

vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
Try to increase from loader.conf.
I'm interesting about vfs.nfsd.v4statelimit.
Verify it by CLI:
# sysctl vfs.nfsd
NAS4Free 10.2.0.2.2115 (x64-embedded), 10.2.0.2.2258 (arm), 10.2.0.2.2258(dom0)
GIGABYTE 5YASV-RH, Celeron E3400 (Dual 2.6GHz), ECC 8GB, Intel ET/CT/82566DM (on-board), ZFS mirror (2TBx2)
ASRock E350M1/USB3, 16GB, Realtek 8111E (on-board), ZFS mirror (2TBx2)
MSI MS-9666, Core i7-860(Quad 2.8GHz/HT), 32GB, Mellanox ConnectX-2 EN/Intel 82578DM (on-board), ZFS mirror (3TBx2+L2ARC/ZIL:SSD128GB)
Develop/test environment:
VirtualBox 512MB VM, ESXi 512MB-8GB VM, Raspberry Pi, Pi2, ODROID-C1

User avatar
tuaris
experienced User
experienced User
Posts: 85
Joined: 19 Jul 2012 21:31
Contact:
Status: Offline

Re: nfsv4 expired locks lost

#3

Post by tuaris »

The current settings are:

Code: Select all

sysctl vfs.nfsd
vfs.nfsd.request_space_throttle_count: 0
vfs.nfsd.request_space_throttled: 0
vfs.nfsd.request_space_low: 31457280
vfs.nfsd.request_space_high: 47185920
vfs.nfsd.request_space_used_highest: 1657432
vfs.nfsd.request_space_used: 4344
vfs.nfsd.groups: 1
vfs.nfsd.threads: 32
vfs.nfsd.maxthreads: 32
vfs.nfsd.minthreads: 32
vfs.nfsd.fha.fhe_stats: No file handle entries.
vfs.nfsd.fha.max_reqs_per_nfsd: 0
vfs.nfsd.fha.max_nfsds_per_fh: 8
vfs.nfsd.fha.bin_shift: 22
vfs.nfsd.fha.enable: 1
vfs.nfsd.cachetcp: 1
vfs.nfsd.tcpcachetimeo: 43200
vfs.nfsd.udphighwater: 500
vfs.nfsd.tcphighwater: 0
vfs.nfsd.enable_stringtouid: 0
vfs.nfsd.debuglevel: 0
vfs.nfsd.enable_locallocks: 0
vfs.nfsd.issue_delegations: 0
vfs.nfsd.commit_miss: 0
vfs.nfsd.commit_blks: 0
vfs.nfsd.mirrormnt: 1
vfs.nfsd.async: 0
vfs.nfsd.server_max_nfsvers: 4
vfs.nfsd.server_min_nfsvers: 2
vfs.nfsd.nfs_privport: 0
vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
vfs.nfsd.enable_nogroupcheck: 1
vfs.nfsd.enable_nobodycheck: 1
vfs.nfsd.disable_checkutf8: 0
Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000

User avatar
daoyama
Developer
Developer
Posts: 422
Joined: 25 Aug 2012 09:28
Location: Japan
Status: Offline

Re: nfsv4 expired locks lost

#4

Post by daoyama »

tuaris wrote: Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000
I don't know. At least, hash size must set before loading kernel.
NAS4Free 10.2.0.2.2115 (x64-embedded), 10.2.0.2.2258 (arm), 10.2.0.2.2258(dom0)
GIGABYTE 5YASV-RH, Celeron E3400 (Dual 2.6GHz), ECC 8GB, Intel ET/CT/82566DM (on-board), ZFS mirror (2TBx2)
ASRock E350M1/USB3, 16GB, Realtek 8111E (on-board), ZFS mirror (2TBx2)
MSI MS-9666, Core i7-860(Quad 2.8GHz/HT), 32GB, Mellanox ConnectX-2 EN/Intel 82578DM (on-board), ZFS mirror (3TBx2+L2ARC/ZIL:SSD128GB)
Develop/test environment:
VirtualBox 512MB VM, ESXi 512MB-8GB VM, Raspberry Pi, Pi2, ODROID-C1

User avatar
tuaris
experienced User
experienced User
Posts: 85
Joined: 19 Jul 2012 21:31
Contact:
Status: Offline

Re: nfsv4 expired locks lost

#5

Post by tuaris »

What value do you recommend I set for the hash size and state limit?
Is their an NFSv4 tuning guide I can read over?

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: nfsv4 expired locks lost

#6

Post by Parkcomm »

Dayoma - could you explain what is causing the problem the changing the vfs.nfsd.v4statelimit solves and how it might come about?

(FYI - I've seen this problem before and ended up "solving" it by falling back to NFSV3 - not ideal)
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

User avatar
tuaris
experienced User
experienced User
Posts: 85
Joined: 19 Jul 2012 21:31
Contact:
Status: Offline

Re: nfsv4 expired locks lost

#7

Post by tuaris »

I've experienced some data loss as a result of this problem. An application running on one of the NFS clients continued to write to a file and the data was never 'saved'.

I've switched back to NFSv3. Looks like NFSv4 isn't ready.

According to this page, this appears to be what's happening.
https://access.redhat.com/solutions/1179643

Post Reply

Return to “NFS”