TierEngine component to deal with tiering, inherits all other components so this is essentially the entirety of TierEngine. More...

#include <tiering.hpp>

Inheritance diagram for TierEngineTiering:

[legend]

Collaboration diagram for TierEngineTiering:

[legend]

Public Member Functions
	TierEngineTiering (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Tiering object. More...

	~TierEngineTiering (void)
	Destroy the Tier Engine Tiering object. More...

void	begin (bool daemon_mode)
	Tier files with tier(), do ad hoc work, and sleep until next period or woken by more work. More...

bool	tier (void)
	Find files, update their popularities, sort the files by popularity, and finally move the files to their respective tiers. Returns true if tiering happened, false if failed to lock mutex. More...

void	launch_crawlers (void(TierEngineTiering::function)(fs::directory_entry &itr, Tier tptr, std::atomic< ffd::Bytes::bytes_type > &usage))
	Call crawl() for each tier in tiers. More...

void	crawl (fs::path dir, Tier tptr, void(TierEngineTiering::function)(fs::directory_entry &itr, Tier *tptr, std::atomic< ffd::Bytes::bytes_type > &usage), std::atomic< ffd::Bytes::bytes_type > &usage)
	Recurse into tier directory, executing function on each file. Function can be emplace_file(), print_file_pins(), or print_file_popularity(). More...

void	emplace_file (fs::directory_entry &file, Tier *tptr, std::atomic< ffd::Bytes::bytes_type > &usage)
	Place file into files_, constructing with fs::path, Tier*, and db_. More...

void	calc_popularity (void)
	Call File::calc_popularity() for each file in files_. More...

void	sort (void)
	Sorts list of files based on popularity, if pop1 == pop2, sort by atime. More...

void	simulate_tier (void)
	Find out which tier each file belongs in. More...

void	move_files (void)
	Launch one thread for each tier to move incoming files into their new backend paths based on results of simulate_tier(). More...

void	update_db (void)
	Iterate over file list and call File::update_db(). More...

void	stop (void)
	Obtain sleep_mt_, set stop_flag_ to true, wake sleeping tier thread with sleep_cv_.notify_one(). Causes sleeping tiering thread to join. More...

bool	currently_tiering (void) const
	Check if currently tiering. More...

bool	strict_period (void) const
	Return config_.strict_period(). More...

void	exit (int status)
	Unlock mutex and call stop() before calling ::exit() More...

Public Member Functions inherited from TierEngineDatabase
	TierEngineDatabase (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Database object Calls open_db(). More...

	~TierEngineDatabase (void)
	Destroy the Tier Engine Database object Deletes (closes) the rocksdb database. More...

std::shared_ptr< rocksdb::DB >	get_db (void)
	Get the database pointer. Used in fusePassthrough.cpp for getting file. More...

Public Member Functions inherited from TierEngineBase
	TierEngineBase (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Base object. More...

virtual	~TierEngineBase (void)
	Destroy the Tier Engine Base object. More...

int	create_run_path (void) const
	Creates path for FIFOs and database, chowning to root:autotier. More...

std::list< Tier > &	get_tiers (void)
	Get reference to the list of tiers.Used in fusePassthrough.cpp to get references to each tier for finding full backend paths. More...

Tier *	tier_lookup (fs::path p)
	Find Tier from path. More...

Tier *	tier_lookup (std::string id)
	Find tier from ID. More...

void	mount_point (const fs::path &mount_point)
	Set the mount_point_ variable. More...

Public Member Functions inherited from TierEngineSleep
	TierEngineSleep (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Sleep object. More...

	~TierEngineSleep (void)
	Destroy the Tier Engine Sleep object. More...

void	sleep_until (std::chrono::steady_clock::time_point t)
	call wait_until on the condition variable. Puts thread to sleep until time reaches t or woken by sleep_cv_.notify_one() More...

void	sleep_until_woken (void)
	call wait on the condition variable. Puts thread to sleep until woken by sleep_cv_.notify_one() More...

Public Member Functions inherited from TierEngineAdhoc
	TierEngineAdhoc (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Adhoc object. More...

	~TierEngineAdhoc ()
	Destroy the Tier Engine Adhoc object. More...

void	set_socket_permissions (void)
	Get the gid for group `autotier` and set the owner of the socket to root:autotier. More...

void	process_adhoc_requests (void)
	Function for ad hoc server thread. Listens to named FIFO in run_path_ to receive ad hoc commands from running autotier while mounted. Grabs command and queues work as an AdHoc object in TierEngineBase::adhoc_work_. More...

void	process_oneshot (const AdHoc &work)
	Enqueue oneshot AdHoc command into adhoc_work_. More...

void	process_pin_unpin (const AdHoc &work)
	Enqueue pin or unpin AdHoc command into adhoc_work_. More...

void	process_status (const AdHoc &work)
	Iterate through list of tiers, printing ID, path, current usage, and watermark. More...

void	process_config (void)
	Dump current configuration settings from memory. More...

void	process_list_pins (void)
	Send all pinned files with the corresponding tier they are pinned to. More...

void	process_list_popularity (void)
	Send all file paths in filesystem along with popularity. More...

void	process_which_tier (AdHoc &work)
	Send table of each argument file along with its corresponding tier name and full backend path. More...

template<typename... Args>
void	enqueue_work (Args &&...args)
	Emplace an AdHoc job into the work queue. More...

void	execute_queued_work (void)
	Process each Adhoc job enqueued in adhoc_queue_, popping them. Called by the tiering thread as part of the main tier loop in begin().

void	pin_files (const std::vector< std::string > &args)
	Iterate through each string in args[1:] to tier name in args[0]. Sets Metadata::pinned_ flag of each file then moves the file. More...

void	unpin_files (const std::vector< std::string > &args)
	Iterate through args, clearing the Metadata::pinned_ flag of each file. More...

void	shutdown_socket_server (void)
	Call socket_server.shutdown() to wake adhoc server thread from wait_for_connection() More...

Public Member Functions inherited from TierEngineMutex
	TierEngineMutex (const fs::path &config_path, const ConfigOverrides &config_overrides)
	Construct a new Tier Engine Mutex object. More...

	~TierEngineMutex (void)
	Destroy the Tier Engine Mutex object, calling unlock_mutex() first. More...

Private Attributes
bool	currently_tiering_
	Whether or not tiering is happening. Set and cleared in tier()

std::chrono::steady_clock::time_point	last_tier_time_
	For determining tier period.

std::vector< File >	files_
	Vector to contain every file across all tiers for sorting.

Additional Inherited Members
Protected Member Functions inherited from TierEngineBase
virtual void	exit (int status) const
	Virtual exit function that can be overridden by other components for cleanup. More...

Protected Member Functions inherited from TierEngineMutex
int	lock_mutex (void)
	Opens file at mutex_path such that if the file already exists, opening fails. Uses this as a mutex lock - if the file exists, the critical section is locked. To unlock, delete the file. More...

void	unlock_mutex (void)
	Deletes mutex lock file, unlocking the critical section. More...

Protected Attributes inherited from TierEngineBase
bool	stop_flag_
	Set to false to make thread exit. Used to continue or cancel sleeping after being woken to do ad hoc command work. More...

std::list< Tier >	tiers_
	List of tiers built from configuration file.

Config	config_
	Configuration read from config_file path.

fs::path	run_path_
	Path to mutex lock file, ad hoc FIFO, and database. Defaults to /var/lib/autotier/<hash of config path>, can be overridden in config.

fs::path	mount_point_
	Where autotier filesystem is mounted, set by mount_point().

ConcurrentQueue< AdHoc >	adhoc_work_

std::condition_variable	sleep_cv_
	Condition variable to use wait_until to sleep between tiering, used for the ad hoc server thread or the main thread to wake the tiering thread from sleep. More...

std::shared_ptr< rocksdb::DB >	db_
	Nosql database holding file metadata.

Protected Attributes inherited from TierEngineSleep
std::mutex	sleep_mt_
	Useless lock to release for the condition_variable to use wait_until.

Protected Attributes inherited from TierEngineMutex
std::mutex	lock_file_mt_
	Used to ensure currently_tiering_ is set atomically with locking the file mutex. More...

Detailed Description

TierEngine component to deal with tiering, inherits all other components so this is essentially the entirety of TierEngine.

Constructor & Destructor Documentation

◆ TierEngineTiering()

TierEngineTiering::TierEngineTiering	(	const fs::path &	config_path,
		const ConfigOverrides &	config_overrides
	)

Construct a new Tier Engine Tiering object.

Parameters

config_path	Path to config file
config_overrides	Config overrides from main()

◆ ~TierEngineTiering()

TierEngineTiering::~TierEngineTiering ( void )

Destroy the Tier Engine Tiering object.

Member Function Documentation

◆ begin()

void TierEngineTiering::begin ( bool daemon_mode )

Tier files with tier(), do ad hoc work, and sleep until next period or woken by more work.

Parameters

daemon_mode Whether or not to daemonise

◆ calc_popularity()

void TierEngineTiering::calc_popularity ( void )

Call File::calc_popularity() for each file in files_.

◆ crawl()

void TierEngineTiering::crawl	(	fs::path	dir,
		Tier *	tptr,
		void(TierEngineTiering::)(fs::directory_entry &itr, Tier tptr, std::atomic< ffd::Bytes::bytes_type > &usage)	function,
		std::atomic< ffd::Bytes::bytes_type > &	usage
	)

Recurse into tier directory, executing function on each file. Function can be emplace_file(), print_file_pins(), or print_file_popularity().

Parameters

dir	Current directory path during recursion
tptr	Pointer to tier it is running for
function	Function to execute on each file
usage	Keeps track of recursive file size for reporting tier usage

◆ currently_tiering()

bool TierEngineTiering::currently_tiering ( void ) const

virtual

Check if currently tiering.

Returns: true Tiering; false Not tiering

Reimplemented from TierEngineBase.

◆ emplace_file()

void TierEngineTiering::emplace_file	(	fs::directory_entry &	file,
		Tier *	tptr,
		std::atomic< ffd::Bytes::bytes_type > &	usage
	)

Place file into files_, constructing with fs::path, Tier*, and db_.

Parameters

file	Directory entry for file, containing path
tptr	Tier the file was found in
usage	Keeps track of recursive file size for reporting tier usage

◆ exit()

void TierEngineTiering::exit ( int status )

Unlock mutex and call stop() before calling ::exit()

Parameters

status Exit status of process

◆ launch_crawlers()

void TierEngineTiering::launch_crawlers ( void(TierEngineTiering::*)(fs::directory_entry &itr, Tier *tptr, std::atomic< ffd::Bytes::bytes_type > &usage) function )

Call crawl() for each tier in tiers.

Parameters

function Function to execute for each tier

◆ move_files()

void TierEngineTiering::move_files ( void )

Launch one thread for each tier to move incoming files into their new backend paths based on results of simulate_tier().

◆ simulate_tier()

void TierEngineTiering::simulate_tier ( void )

Find out which tier each file belongs in.

Pseudocode: LET file = first file of sorted files LET tier = first tier LET tier_usage = size of pinned files WHILE file != last file of sorted files DO IF tier_usage + size of file > tier watermark AND tier != last tier THEN tier = next tier tier_usage = 0 END IF file belongs to tier (push file into tier.incoming_files_) tier_usage += size of file file = next file END DO

◆ sort()

void TierEngineTiering::sort ( void )

Sorts list of files based on popularity, if pop1 == pop2, sort by atime.

◆ stop()

void TierEngineTiering::stop ( void )

Obtain sleep_mt_, set stop_flag_ to true, wake sleeping tier thread with sleep_cv_.notify_one(). Causes sleeping tiering thread to join.

◆ strict_period()

bool TierEngineTiering::strict_period ( void ) const

Return config_.strict_period().

Returns: true Tier period is strict (no automatic tiers); false Tiering can happen whenever needed

◆ tier()

bool TierEngineTiering::tier ( void )

virtual

Find files, update their popularities, sort the files by popularity, and finally move the files to their respective tiers. Returns true if tiering happened, false if failed to lock mutex.

Returns: true Tiered successfully; false Failed to lock mutex

Reimplemented from TierEngineBase.

◆ update_db()

void TierEngineTiering::update_db ( void )

Iterate over file list and call File::update_db().

The documentation for this class was generated from the following files:

src/incl/TierEngine/components/tiering.hpp
src/impl/autotierfs/TierEngine/components/tiering.cpp

Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ TierEngineTiering()

◆ ~TierEngineTiering()

Member Function Documentation

◆ begin()

◆ calc_popularity()

◆ crawl()

◆ currently_tiering()

◆ emplace_file()

◆ exit()

◆ launch_crawlers()

◆ move_files()

◆ simulate_tier()

◆ sort()

◆ stop()

◆ strict_period()

◆ tier()

◆ update_db()