How many files in a zfs filesystem

The usable space is 1 TB. When the 1 TB drive is replaced with another 2 TB drive, the resilvering process copies the existing data onto the new drive.

Expansion is triggered by using zpool online -e on each device. After expansion of all devices, the additional space becomes available to the pool. Pools are exported before moving them to another system. All datasets are unmounted, and each device is marked as exported but still locked so it cannot be used by other disk subsystems. This allows pools to be imported on other machines, other operating systems that support ZFS, and even different hardware architectures with some caveats, see zpool 8.

When a dataset has open files, zpool export -f can be used to force the export of a pool. Use this with caution. The datasets are forcibly unmounted, potentially resulting in unexpected behavior by the applications which had open files on those datasets. Importing a pool automatically mounts the datasets. This may not be the desired behavior, and can be prevented with zpool import -N. If the pool was last used on a different system and was not properly exported, an import might have to be forced with zpool import -f.

After upgrading FreeBSD, or if a pool has been imported from a system using an older version of ZFS, the pool can be manually upgraded to the latest version of ZFS to support newer features. Consider whether the pool may ever need to be imported on an older system before upgrading.

Upgrading is a one-way process. Older pools can be upgraded, but pools with newer features cannot be downgraded. The newer features of ZFS will not be available until zpool upgrade has completed. The boot code on systems that boot from a pool must be updated to support the new pool version. Use gpart bootcode on the partition that contains the boot code.

There are two types of bootcode available, depending on way the system boots: GPT the most common option and EFI for more modern systems.

Apply the bootcode to all bootable disks in the pool. See gpart 8 for more information. Commands that modify the pool are recorded. Recorded actions include the creation of datasets, changing properties, or replacement of a disk. This history is useful for reviewing how a pool was created and which user performed a specific action and when.

History is not kept in a log file, but is part of the pool itself. The command to review this history is aptly named zpool history :. The output shows zpool and zfs commands that were executed on the pool along with a timestamp. Only commands that alter the pool in some way are recorded. Commands like zfs list are not included. When no pool name is specified, the history of all pools is displayed.

More details can be shown by adding -l. History records are shown in a long format, including information like the name of the user who issued the command and the hostname on which the change was made. The hostname display becomes important when the pool is exported from one system and imported on another.

The commands that are issued on the other system can clearly be distinguished by the hostname that is recorded for each command. Both options to zpool history can be combined to give the most detailed information possible for any given pool.

Pool history provides valuable information when tracking down the actions that were performed or when more detailed output is needed for debugging. By default, all pools in the system are monitored and displayed. A pool name can be provided to limit monitoring to just that pool. A basic example:. The next statistic line is printed after each interval.

Alternatively, give a second number on the command line after the interval to specify the total number of statistics to display. Each device in the pool is shown with a statistics line.

This is useful in seeing how many read and write operations are being performed on each device, and can help determine if any individual device is slowing down the pool.

This example shows a mirrored pool with two devices:. A pool consisting of one or more mirror vdevs can be split into two pools. Unless otherwise specified, the last member of each mirror is detached and used to create a new pool containing the same data. The operation should first be attempted with -n. The details of the proposed operation are displayed without it actually being performed. This helps confirm that the operation will do what the user intends.

The zfs utility is responsible for creating, destroying, and managing all ZFS datasets that exist within a pool. The pool is managed using zpool. Unlike traditional disks and volume managers, space in ZFS is not preallocated.

With traditional file systems, after all of the space is partitioned and assigned, there is no way to add an additional file system without adding a new disk.

With ZFS, new file systems can be created at any time. Each dataset has properties including features like compression, deduplication, caching, and quotas, as well as other useful properties like readonly, case sensitivity, network file sharing, and a mount point.

Datasets can be nested inside each other, and child datasets will inherit properties from their parents. Each dataset can be administered, delegated , replicated , snapshotted , jailed , and destroyed as a unit.

There are many advantages to creating a separate dataset for each different type or set of files. The only drawbacks to having an extremely large number of datasets is that some commands like zfs list will be slower, and the mounting of hundreds or even thousands of datasets can slow the FreeBSD boot process. Destroying a dataset is much quicker than deleting all of the files that reside on the dataset, as it does not involve scanning all of the files and updating all of the corresponding metadata.

In modern versions of ZFS, zfs destroy is asynchronous, and the free space might take several minutes to appear in the pool. Use zpool get freeing poolname to see the freeing property, indicating how many datasets are having their blocks freed in the background. If there are child datasets, like snapshots or other datasets, then the parent cannot be destroyed. To destroy a dataset and all of its children, use -r to recursively destroy the dataset and all of its children.

Use -n -v to list datasets and snapshots that would be destroyed by this operation, but do not actually destroy anything. Space that would be reclaimed by destruction of snapshots is also shown. A volume is a special type of dataset. This allows the volume to be used for other file systems, to back the disks of a virtual machine, or to be exported using protocols like iSCSI or HAST. A volume can be formatted with any file system, or used without a file system to store raw data.

To the user, a volume appears to be a regular disk. Putting ordinary file systems on these zvols provides features that ordinary disks or file systems do not normally have. For example, using the compression property on a MB volume allows creation of a compressed FAT file system. Destroying a volume is much the same as destroying a regular file system dataset. The operation is nearly instantaneous, but it may take several minutes for the free space to be reclaimed in the background.

The name of a dataset can be changed with zfs rename. The parent of a dataset can also be changed with this command. Renaming a dataset to be under a different parent dataset will change the value of those properties that are inherited from the parent dataset. When a dataset is renamed, it is unmounted and then remounted in the new location which is inherited from the new parent dataset.

This behavior can be prevented with -u. Snapshots can also be renamed like this. Due to the nature of snapshots, they cannot be renamed into a different parent dataset. To rename a recursive snapshot, specify -r , and all snapshots with the same name in child datasets will also be renamed.

Each ZFS dataset has a number of properties that control its behavior. Most properties are automatically inherited from the parent dataset, but can be overridden locally. Most properties have a limited set of valid values, zfs get will display each possible property and valid values.

Most properties can be reverted to their inherited values using zfs inherit. User-defined properties can also be set. They become part of the dataset configuration and can be used to provide additional information about the dataset or its contents. To distinguish these custom properties from the ones supplied as part of ZFS, a colon : is used to create a custom namespace for the property. To remove a custom property, use zfs inherit with -r. Setting these define if and how ZFS datasets may be shared on the network.

To get the current status of a share, enter:. It is also possible to set additional options for sharing datasets through NFS, such as -alldirs , -maproot and -network. To set additional options to a dataset shared through NFS, enter:. Snapshots are one of the most powerful features of ZFS. A snapshot provides a read-only, point-in-time copy of the dataset. If no snapshots exist, space is reclaimed for future use when data is rewritten or deleted.

Snapshots preserve disk space by recording only the differences between the current dataset and a previous version. Snapshots are allowed only on whole datasets, not on individual files or directories. When a snapshot is created from a dataset, everything contained in it is duplicated. This includes the file system properties, files, directories, permissions, and so on. Snapshots use no additional space when they are first created, only consuming space as the blocks they reference are changed.

Recursive snapshots taken with -r create a snapshot with the same name on the dataset and all of its children, providing a consistent moment-in-time snapshot of all of the file systems.

This can be important when an application has files on multiple datasets that are related or dependent upon each other. Without snapshots, a backup would have copies of the files from different points in time. Snapshots in ZFS provide a variety of features that even other file systems with snapshot functionality lack. A typical example of snapshot use is to have a quick way of backing up the current state of the file system when a risky action like a software installation or a system upgrade is performed.

If the action fails, the snapshot can be rolled back and the system has the same state as when the snapshot was created. If the upgrade was successful, the snapshot can be deleted to free up space. Without snapshots, a failed upgrade often requires a restore from backup, which is tedious, time consuming, and may require downtime during which the system cannot be used.

Snapshots can be rolled back quickly, even while the system is running in normal operation, with little or no downtime. The time savings are enormous with multi-terabyte storage systems and the time required to copy the data from backup. Snapshots are not a replacement for a complete backup of a pool, but can be used as a quick and easy way to store a copy of the dataset at a specific point in time.

Snapshots are created with zfs snapshot dataset snapshotname. Adding -r creates a snapshot recursively, with the same name on all child datasets. Snapshots are not shown by a normal zfs list operation. To list snapshots, -t snapshot is appended to zfs list.

There is no mention of available disk space in the AVAIL column, as snapshots cannot be written to after they are created. Compare the snapshot to the original dataset from which it was created:. Displaying both the dataset and the snapshot together reveals how snapshots work in COW fashion.

They save only the changes delta that were made and not the complete file system contents all over again. This means that snapshots take little space when few changes are made. Space usage can be made even more apparent by copying a file to the dataset, then making a second snapshot:. The second snapshot contains only the changes to the dataset after the copy operation.

This yields enormous space savings. ZFS provides a built-in command to compare the differences in content between two snapshots. This is helpful when many snapshots were taken over time and the user wants to see how the file system has changed over time. For example, zfs diff lets a user find the latest snapshot that still contains a file that was accidentally deleted.

Doing this for the two snapshots that were created in the previous section yields this output:. The first column shows the type of change:. Comparing two snapshots is helpful when using the ZFS replication feature to transfer a dataset to a different host for backup purposes.

A backup administrator can compare two snapshots received from the sending host and determine the actual changes in the dataset. See the Replication section for more information. When at least one snapshot is available, it can be rolled back to at any time. Most of the time this is the case when the current state of the dataset is no longer required and an older version is preferred. Luckily, rolling back a snapshot is just as easy as typing zfs rollback snapshotname.

Depending on how many changes are involved, the operation will finish in a certain amount of time. During that time, the dataset always remains in a consistent state, much like a database that conforms to ACID principles is performing a rollback. This is happening while the dataset is live and accessible without requiring a downtime.

Once the snapshot has been rolled back, the dataset has the same state as it had when the snapshot was originally taken. All other data in that dataset that was not part of the snapshot is discarded. Taking a snapshot of the current state of the dataset before rolling back to a previous one is a good idea when some data is required later. This way, the user can roll back and forth between snapshots without losing data that is still valuable.

In the first example, a snapshot is rolled back because of a careless rm operation that removes too much data than was intended. At this point, the user realized that too many files were deleted and wants them back. ZFS provides an easy way to get them back using rollbacks, but only when snapshots of important data are performed on a regular basis.

To get the files back and start over from the last snapshot, issue the command:. The rollback operation restored the dataset to the state of the last snapshot. It is also possible to roll back to a snapshot that was taken much earlier and has other snapshots that were created after it. When trying to do this, ZFS will issue this warning:. This warning means that snapshots exist between the current state of the dataset and the snapshot to which the user wants to roll back.

To complete the rollback, these snapshots must be deleted. ZFS cannot track all the changes between different states of the dataset, because snapshots are read-only. ZFS will not delete the affected snapshots unless the user specifies -r to indicate that this is the desired action.

If that is the intention, and the consequences of losing all intermediate snapshots is understood, the command can be issued:. The output from zfs list -t snapshot confirms that the intermediate snapshots were removed as a result of zfs rollback -r. Snapshots are mounted in a hidden directory under the parent dataset:. By default, these directories will not be displayed even when a standard ls -a is issued.

Although the directory is not displayed, it is there nevertheless and can be accessed like any normal directory. The property named snapdir controls whether these hidden directories show up in a directory listing. Setting the property to visible allows them to appear in the output of ls and other commands that deal with directory contents. Individual files can easily be restored to a previous state by copying them from the snapshot back to the parent dataset.

The directory structure below. In the next example, it is assumed that a file is to be restored from the hidden. When ls. It is up to the administrator to decide whether these directories will be displayed. It is possible to display these for certain datasets and prevent it for others.

Copying files or directories from this hidden. Trying it the other way around results in this error:. The error reminds the user that snapshots are read-only and cannot be changed after creation. Files cannot be copied into or removed from snapshot directories because that would change the state of the dataset they represent. Snapshots consume space based on how much the parent file system has changed since the time of the snapshot.

The written property of a snapshot tracks how much space is being used by the snapshot. Snapshots are destroyed and the space reclaimed with zfs destroy dataset snapshot. Adding -r recursively removes all snapshots with the same name under the parent dataset.

Adding -n -v to the command displays a list of the snapshots that would be deleted and an estimate of how much space would be reclaimed without performing the actual destroy operation. A clone is a copy of a snapshot that is treated more like a regular dataset. Unlike a snapshot, a clone is not read only, is mounted, and can have its own properties.

Once a clone has been created using zfs clone , the snapshot it was created from cannot be destroyed. After a clone has been promoted, the snapshot becomes a child of the clone, rather than of the original parent dataset.

This will change how the space is accounted, but not actually change the amount of space consumed. The clone can be mounted at any point within the ZFS file system hierarchy, not just below the original location of the snapshot. A typical use for clones is to experiment with a specific dataset while keeping the snapshot around to fall back to in case something goes wrong. After the desired result is achieved in the clone, the clone can be promoted to a dataset and the old file system removed.

This is not strictly necessary, as the clone and dataset can coexist without problems. After a clone is created it is an exact copy of the state the dataset was in when the snapshot was taken. The clone can now be changed independently from its originating dataset. The only connection between the two is the snapshot. ZFS records this connection in the property origin. Once the dependency between the snapshot and the clone has been removed by promoting the clone using zfs promote , the origin of the clone is removed as it is now an independent dataset.

This example demonstrates it:. After making some changes like copying loader. Instead, the promoted clone can replace it. This can be achieved by two consecutive commands: zfs destroy on the old dataset and zfs rename on the clone to name it like the old dataset it could also get an entirely different name. The cloned snapshot is now handled like an ordinary dataset. It contains all the data from the original snapshot plus the files that were added to it like loader. Clones can be used in different scenarios to provide useful features to ZFS users.

For example, jails could be provided as snapshots containing different sets of installed applications. Users can clone these snapshots and add their own applications as they see fit.

Once they are satisfied with the changes, the clones can be promoted to full datasets and provided to end users to work with like they would with a real dataset. This saves time and administrative overhead when providing these jails. Keeping data on a single pool in one location exposes it to risks like theft and natural or human disasters. Making regular backups of the entire pool is vital.

ZFS provides a built-in serialization feature that can send a stream representation of the data to standard output. Using this technique, it is possible to not only store the data on another pool connected to the local system, but also to send it over a network to another system.

Snapshots are the basis for this replication see the section on ZFS snapshots. The commands used for replicating data are zfs send and zfs receive. The pool named mypool is the primary pool where data is written to and read from on a regular basis.

A second pool, backup is used as a standby in case the primary pool becomes unavailable. Note that this fail-over is not done automatically by ZFS, but must be manually done by a system administrator when needed. A snapshot is used to provide a consistent version of the file system to be replicated. Once a snapshot of mypool has been created, it can be copied to the backup pool.

Only snapshots can be replicated. Changes made since the most recent snapshot will not be included. Now that a snapshot exists, zfs send can be used to create a stream representing the contents of the snapshot. This stream can be stored as a file or received by another pool. The stream is written to standard output, but must be redirected to a file or pipe or an error is produced:. To back up a dataset with zfs send , redirect to a file located on the mounted backup pool. Ensure that the pool has enough free space to accommodate the size of the snapshot being sent, which means all of the data contained in the snapshot, not just the changes from the previous snapshot.

The zfs send transferred all the data in the snapshot called backup1 to the pool named backup. Creating and sending these snapshots can be done automatically with a cron 8 job. Instead of storing the backups as archive files, ZFS can receive them as a live file system, allowing the backed up data to be accessed directly. To get to the actual data contained in those streams, zfs receive is used to transform the streams back into files and directories.

The example below combines zfs send and zfs receive using a pipe to copy the data from one pool to another. The data can be used directly on the receiving pool after the transfer is complete. A dataset can only be replicated to an empty dataset.

This saves disk space and transfer time. For example:. A second snapshot called replica2 was created. This second snapshot contains only the changes that were made to the file system between now and the previous snapshot, replica1. Using zfs send -i and indicating the pair of snapshots generates an incremental replica stream containing only the data that has changed.

This can only succeed if the initial snapshot already exists on the receiving side. The incremental stream was successfully transferred. Only the data that had changed was replicated, rather than the entirety of replica1. Only the differences were sent, which took much less time to transfer and saved disk space by not copying the complete pool each time.

This is useful when having to rely on slow networks or when costs per transferred byte must be considered. If -P is specified, the properties of the dataset will be copied, including compression settings, quotas, and mount points. When -R is specified, all child datasets of the indicated dataset will be copied, along with all of their properties. Sending and receiving can be automated so that regular backups are created on the second pool.

Sending streams over the network is a good way to keep a remote backup, but it does come with a drawback.

Data sent over the network link is not encrypted, allowing anyone to intercept and transform the streams back into data without the knowledge of the sending user. This is undesirable, especially when sending the streams over the internet to a remote host. SSH can be used to securely encrypt data send over a network connection. Since ZFS only requires the stream to be redirected from standard output, it is relatively easy to pipe it through SSH. To keep the contents of the file system encrypted in transit and on the remote system, consider using PEFS.

A few settings and security precautions must be completed first. Only the necessary steps required for the zfs send operation are shown here. Normally, the privileges of the root user are needed to send and receive streams. This requires logging in to the receiving system as root. However, logging in as root is disabled by default for security reasons. The ZFS Delegation system can be used to allow a non- root user on each system to perform the respective send and receive operations.

To mount the pool, the unprivileged user must own the directory, and regular users must be allowed to mount file systems. On the receiving system:. The unprivileged user now has the ability to receive and mount datasets, and the home dataset can be replicated to the remote system:. A recursive snapshot called monday is made of the file system dataset home that resides on the pool mypool. Then it is sent with zfs send -R to include the dataset, all child datasets, snapshots, clones, and settings in the stream.

The output is piped to the waiting zfs receive on the remote host backuphost through SSH. Using a fully qualified domain name or IP address is recommended. The receiving machine writes the data to the backup dataset on the recvpool pool.

Adding -d to zfs recv overwrites the name of the pool on the receiving side with the name of the snapshot. When -v is included, more detail about the transfer is shown, including elapsed time and the amount of data transferred. Dataset quotas are used to restrict the amount of space that can be consumed by a particular dataset. Reference Quotas work in very much the same way, but only count the space used by the dataset itself, excluding snapshots and child datasets.

Similarly, user and group quotas can be used to prevent users or groups from using all of the space in the pool or dataset. The following examples assume that the users already exist in the system. This will properly set owner and group permissions without shadowing any pre-existing home directory paths that might exist. User quota properties are not displayed by zfs get all. Non- root users can only see their own quotas unless they have been granted the userquota privilege.

To remove the quota for the group firstgroup , or to make sure that one is not set, instead use:. As with the user quota property, non- root users can only see the quotas associated with the groups to which they belong. However, root or a user with the groupquota privilege can view and set all quotas for all groups.

To display the amount of space used by each user on a file system or snapshot along with any quotas, use zfs userspace. For group information, use zfs groupspace. For more information about supported options or how to display only specific options, refer to zfs 1. Quantities are limited. With the launch of Chia and a new wave of cryptocurrency growth, storage demands have soared and capacity is a major concern.

As insignificant as it sounds, there is a lot of responsibility that comes with being the last word in file systems. The distinctions between the two file system types lie in how they both handle original blocks of data after duplicating them. The copy-on-write file system overwrites original data blocks after reproducing an exact copy in a new location. On the other hand, the redirect-on-write file system particularly the ZFS file system updates the metadata so that it points—redirects—towards the newly written blocks.

By design, ZFS is capable of managing hundreds and even thousands of network-attached storage NAS drives on one network. It can do this on a solitary network while pooling all available storage as though it existed within one huge NAS drive.

For this reason, ZFS file systems are highly scalable, supporting large maximum file sizes; if the system requires additional capacity, users only need to add more drives to the network. ZFS and OpenZFS file systems are capable of producing point-in-time copies of file systems quite fast and efficiently since the system preserves all copies of the data. Snapshots are considered immutable copies data is replaced rather than changed , while clones are mutable data is modified and updated in-place regularly.

With ZFS on Solaris, snapshots and clones are integrated into boot environments and allow users to conduct a rollback of previous snapshots in the event something goes wrong while updating or patching a system. Lastly, ZFS snapshots and clones are beneficial as a recovery tool in the event of a ransomware attack. This article has sung the benefits of ZFS.

Now let me tell you a quick problem with ZFS. Using RAID-Z can be expensive because of the number of drives you need to purchase to add storage space.

If you found this article interesting, please take a minute to share it on social media, Hacker News or Reddit. My name is John Paul Wohlscheid. I'm an aspiring mystery writer who loves to play with technology, especially Linux. You can catch up with me at my personal website. You can check out my ebooks here. I also write a newsletter about the stuff that most history books miss.

Check it out. I just went through a night mare with LVM2 shared from a Linux server to mac books. It would not create the full size of the entire storage pool i.

I installed ZFS and configuration was a snap, the entire 7TB was configure without a problem and performance surpasswd LVM2 by copying the same amount of data in just 10 minutes.

Hats of to the ZFS development team as the save my behind and lack of sleep as it is currently am. Nice article. Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page. Why are People Crazy About it? Pooled storage in ZFS. Like what you read?

Please share it with others. Thank you very much ZFS team Hats off. Tired Herb ;. Policies Affiliate Policy Privacy Policy.