liberata_metrics.utils package
Overview
The utils module provides essential utilities for working with the Liberata Scientometrics library. It includes:
Data Loading: Fetch data from Supabase or local files
Data Wrangling: Transform and prepare data for analysis
Matrix Operations: Efficient utilities for sparse matrix manipulation
Helper Functions: Common operations used across the package
Submodules
Common Tasks
Loading Data from Supabase
from liberata_metrics.utils import load_supabase_data
# Load references and capital matrices from production
references, capital = load_supabase_data.fetch_matrices()
# Load specific time period
start_date = '2023-01-01'
end_date = '2024-01-01'
refs_period, cap_period = load_supabase_data.fetch_matrices_for_period(
start_date, end_date
)
Loading from Local Files
from liberata_metrics.utils import data_loading
from scipy import sparse
import numpy as np
# Load COO matrices
references = data_loading.load_coo_matrix('data/references.npz')
capital = data_loading.load_coo_matrix('data/capital.npz')
# Load metadata
id_maps = data_loading.load_id_mappings('data/mappings.json')
Data Wrangling
from liberata_metrics.utils import data_wrangling
# Filter matrix for a subset of manuscripts
subset_references = data_wrangling.filter_manuscripts(
references,
manuscript_ids
)
# Normalize capital allocation
normalized_capital = data_wrangling.normalize_capital(capital)
# Aggregate by topic
topic_aggregated = data_wrangling.aggregate_by_topic(
capital,
topic_mappings
)
Sparse Matrix Operations
from liberata_metrics.utils import utils
import numpy as np
# Safe division on sparse matrices
result = utils.sparse_divide(numerator, denominator)
# Extract submatrix
sub = utils.submatrix(matrix, row_indices, col_indices)
# Convert between sparse formats
lil_matrix = utils.to_lil(coo_matrix)
csr_matrix = utils.to_csr(coo_matrix)
Performance Considerations
Working with Large Matrices
The utils module is optimized for large sparse matrices:
Sparse matrix format (COO, CSR, CSC) to minimize memory usage
Lazy operations that avoid creating dense copies
Vectorized NumPy operations for speed
from liberata_metrics.utils import data_loading
import scipy.sparse as sparse
# Load large matrix efficiently
large_matrix = data_loading.load_coo_matrix('data/large.npz')
# Convert to efficient format for operations
csr_matrix = large_matrix.tocsr() # For row operations
csc_matrix = large_matrix.tocsc() # For column operations
# Avoid creating dense copies
result = csr_matrix.multiply(another_sparse) # Stays sparse
Memory Profiling
from liberata_metrics.utils import utils
import sys
# Check size of matrix in memory
size_mb = sys.getsizeof(matrix) / 1e6
nnz = matrix.nnz # Number of non-zero elements
density = nnz / (matrix.shape[0] * matrix.shape[1])
Module Contents
- liberata_metrics.utils.coo_to_binned_array(coo: coo_matrix, max_side: int, agg_fn: Callable = <function sum>) ndarray[source]
utility function to convert COO to binned array
- liberata_metrics.utils.deserialize_upload_dates(upload_dates_json: Dict[str, str]) Dict[str, date][source]
convert ISO strings YYYY-MM-DD to date objects
- liberata_metrics.utils.make_date_grid(start_date: date, end_date: date, time_step: timedelta) List[date][source]
build a grid of dates from start_date to end_date with given time_step increments for time series data extraction
- liberata_metrics.utils.matrix_to_plot_array(mat: spmatrix, max_side: int = 500, agg_fn: Callable = <function sum>) ndarray[source]
convert any COO matrix into appropriate array form for plotting (note: this is all ChatGPT)
- liberata_metrics.utils.random_date(rng: RandomState, start: date, end: date) date[source]
draw a random date in given range
- liberata_metrics.utils.read_yaml_config(config_path: str)[source]
reads YAML config file and returns dictionary