The Data Tightrope: Where Accessibility Meets Privacy

Let’s face it—in today’s data landscape, data is simultaneously your most valuable asset and your biggest potential liability.

Finding that sweet spot where data remains accessible enough to drive business decisions while being locked down enough to satisfy privacy regulations. It’s not just about ticking compliance boxes—it’s about maintaining customer trust while still extracting every bit of analytical value from your data assets.



The Four-Tier Data Storage Reality

Most financial institutions operate with multiple data storage options, each with its own tradeoffs. Here’s how I’ve seen it play out in practice:

  1. Glacier/Cold Storage: Dirt cheap but painfully slow—we’re talking potentially days to retrieve data. Perfect for those regulatory archival requirements where immediacy isn’t essential.

  2. Iceberg Relational Databases: Your traditional SQL databases offering a balance of speed and cost. This is where most of your operational data lives.

  3. In-memory Caching Solutions (Redis, Memcached): Lightning-fast but expensive. Reserve these for those mission-critical nanosecond accesses.

  4. Hybrid Solutions (like Druid): Offering column-store capabilities with better performance than traditional databases but lower cost than pure in-memory solutions.

Each tier has its place in your architecture, but—and this is crucial—each needs its own privacy controls. I’ve seen too many organizations focus all their privacy efforts on relational databases while neglecting controls on their cold storage or caching layers. Remember Facebook’s Cambridge Analytica debacle? That wasn’t a failure of their primary database privacy controls; it was a failure in their broader data ecosystem.



Anonymization vs. Pseudonymization: More Than Just Semantics

Here’s where things get interesting—and where I’ve seen many companies stumble. The distinction between anonymization and pseudonymization isn’t just academic; it fundamentally affects how you can use your data and what regulatory requirements apply.

Anonymization: The One-Way Trip

Truly anonymized data is aggregated to the point where it cannot be traced back to an individual. Take our retail banking metrics dashboard showing “170 million active accounts in Australia”—valuable for trend analysis but impossible to connect to any specific customer.

But anonymization has limitations. I learned this the hard way when we created what we thought was an anonymized dashboard. By slicing the data across too many dimensions (location, account type, account age, balance tier), we inadvertently created a situation where some data points could be traced back to specific high-net-worth individuals. The fix? Adding minimum thresholds for each dimensional cut to prevent de-anonymization.

Pseudonymization: Maintaining the Bridge

Pseudonymization differs fundamentally in that it maintains a separate mapping table linking the scrubbed identifiers to the original data. Replace customer IDs with randomly generated tokens while maintaining a secured mapping table accessible only to authorized personnel.

This approach gives you more analytical flexibility but requires rigorous governance of that mapping table. The moment that mapping table becomes compromised, all pseudonymized data effectively becomes identified again.



Practical Implementation

Let me walk you through a example. We created an anonymized version of sport player data as a proxy for customer financial data. Here’s how we structured it:

  • Player Name (representing our customer identifier)
  • Anonymized ID (a bigint with no connection to the original ID)
  • Last Active Season (similar to “last transaction date”)
  • Current Season (representing the present time benchmark)

The anonymization policy stated that players inactive for five years or more would be completely anonymized. This mirrors our policy where customer data achieves a different privacy status after extended inactivity.

Here’s what the implementation looked like:

-- Create the anonymized table
CREATE TABLE anonymized_players (
    anonymized_player_ID INT,
    start_position VARCHAR,
    points DOUBLE,
    season INT,
    game_ID INT,
    PRIMARY KEY (anonymized_player_ID)
);

-- Partition for performance
ALTER TABLE anonymized_players
PARTITION BY RANGE (season);

-- Identify players for anonymization
WITH combined AS (
    -- Join original and anonymized tables
    SELECT
        p.player_name,
        a.anonymized_id,
        p.last_active_season,
        EXTRACT(YEAR FROM CURRENT_DATE) as current_season
    FROM players p
    LEFT JOIN anonymized_players a ON p.player_id = a.original_id
)
-- Apply the anonymization policy
INSERT INTO anonymized_players (player_name, anonymized_id)
SELECT player_name, anonymized_id
FROM combined
WHERE last_active_season < current_season - 5;

The real magic happens with those time-based policies. We have to carefully track player (customer) returns and maintain consistent anonymized IDs. Think about Michael Jordan’s comeback—if he retired in 1998 and returned in 2001, he’d maintain the same anonymized ID because he didn’t exceed our five-year threshold. If he’d stayed retired for six years, he’d receive a completely new anonymized ID upon return.



The Sliding Window Challenge

This sliding window concept creates fascinating technical challenges. We need to:

  1. Maintain rolling retention policies that automatically anonymize data after the specified period
  2. Handle returns gracefully without compromising privacy
  3. Ensure consistency across all data tiers

One approach we’ve found effective is using Iceberg tables with time travel capabilities. This allows us to maintain a view of our table history while only serving the current state to clients.

-- The simplified approach to identifying last partition update
SELECT
    partition_key,
    MAX(committed_at) as last_update
FROM iceberg_partitions
WHERE partition_key = 'season=2003'
GROUP BY partition_key;


Retention Policies: Logical vs. Creation Date

The retention policy headache doesn’t end with anonymization. You also need to decide whether to base your retention on:

  1. Logical Date: Tied to when the event actually occurred
  2. Creation Date: When the data was loaded into your warehouse

This seemingly minor distinction creates massive downstream implications. Let me illustrate:

Imagine you have a 50-year-old partition (data from 1975). With a logical date retention policy, that partition might be deleted tomorrow because it exceeds your retention window. With a creation date policy, if you just loaded that data last week, it would be retained regardless of how old the actual data is.

My recommendation? Use logical date retention for operational data and creation date for historical archives. This gives you the best of both worlds—efficient purging of operational data while maintaining valuable historical information.



Implementing Data Minimization in Practice

Data minimization isn’t just theoretical—here’s how we’ve implemented it:

  1. Identify Low-Value Data: We analyze query patterns to identify rarely accessed data that can be archived or removed.

  2. Regular Audits: Schedule quarterly reviews of storage and retention policies.

  3. Efficient Data Modeling: Design schemas to minimize redundancy while maximizing query performance.

  4. Compression Strategies: Apply appropriate compression techniques based on access patterns.

Some organizations, like Facebook, have sophisticated tools like “Cockpit” to analyze data usage patterns. While most of us don’t have such advanced tooling, we can implement similar practices at a smaller scale:

-- Simple query to identify rarely used tables
SELECT
    table_name,
    last_query_date,
    DATEDIFF(day, last_query_date, CURRENT_DATE) as days_since_last_access
FROM table_access_logs
ORDER BY days_since_last_access DESC;

The beauty of this approach is that it helps us maintain compliance with data minimization principles in privacy regulations while also improving our operational efficiency.

The Special Case of Active Investigations

Data privacy isn’t always straightforward. What happens when a customer requests deletion, but their data is needed for an active fraud investigation? These edge cases require careful consideration and clear policies.

We’ve implemented a special retention flag in our system that can override standard deletion processes when legal requirements necessitate it. This ensures we maintain compliance with both privacy regulations and legal obligations.

-- Simplified example of deletion logic with legal hold exceptions
DELETE FROM customer_data
WHERE
    customer_id = ?
    AND NOT EXISTS (
        SELECT 1
        FROM legal_holds
        WHERE customer_id = customer_data.customer_id
        AND is_active = true
    );


Wrapping Up: The Human Element

At the end of the day, data management isn’t just a technical challenge—it’s a human one. The electricity consumed by our data operations, the privacy of our customers, and the insights we generate all have real-world impacts on people’s lives.

As data professionals in the financial sector, we have a responsibility to make conscious choices about how we manage data. By prioritizing data efficiency, respecting privacy, and fostering a culture of awareness within our organizations, we can contribute to a more sustainable and ethical data ecosystem.

I’d love to hear about your experiences balancing data accessibility and privacy in your organization. What challenges have you faced, and what solutions have you implemented? Let’s continue this conversation in the comments below.

Remember, every bit of data we manage represents a real person with real privacy concerns. Let’s never lose sight of that as we navigate the complex world of data engineering in financial services.