Sunbird RC
v0.0.14
v0.0.14
  • Learn
    • Introduction
    • Verifiable Credentials
      • What issues will Verifiable Credentials address?
      • What are the key roles in Verifiable Credentials?
      • What are the components of Verifiable Credentials?
      • What are the benefits of Verifiable Credentials?
      • Digital Credentials vs Verifiable Credentials
      • QR code vs Verifiable QR code
      • Use Cases
    • Electronic Registries
      • Evolution of Electronic Registries
      • What issues will Electronic Registries address?
      • Benefits of Electronic Registries
      • Registry vs Database
      • Design Principles
      • Use Cases
    • Sunbird RC Overview
      • Features
      • Core Registry Verbs
      • Why do we need Sunbird RC?
      • What Sunbird RC is and what it's not? (WIP)
      • Core Capabilities
      • Technical Specification Draft
      • Workflows
      • High level architecture
    • Sunbird RC in action
      • Implementations (Work in Progress)
      • Possibilities
  • Use
    • Technical Requirements
    • Releases
    • Setup the Backend
    • Setup the Frontend
    • Leveraging Existing data stores
    • SSO with existing systems
      • Digilocker Meripehchaan SSO
  • Developer Documentation
    • Installation Guide
      • Registry CLI
        • Setup A Registry Instance
      • Manual installation through docker-compose
      • Production setup through Helm
    • Introduction To Schemas
    • Creating Your Own Schemas
    • Schema Configuration
    • Using The APIs
    • Create Schemas With Custom Password
    • Admin Portal
      • Login
      • Get Started
        • Create Schema
        • Attestation Workflows (WIP)
        • VC Template
          • Custom VC Template (WIP)
        • Ownership (WIP)
        • Publish (WIP)
      • Dashboard
    • Configuration
    • Developer Setup
    • VC Verification Module
    • Audit Configuration
    • Custom Keycloak Build
    • Metrics
    • Digilocker Integration
    • Custom QR Code design
    • Notifications Configuration
    • View Templates Configuration
    • Generic Identity And Access Management
    • Backup and Restore
      • PostgreSQL
        • SQL Dump
        • File System Level Backup
        • Continuous Archiving and Point-in-Time Recovery (PITR)
      • Cassandra
        • Snapshot-based backup method
        • Incremental backup method
        • Data Restore
    • Frontend Configurations
    • Frontend - Proxy configuration
  • API Reference
    • Registry
      • Create An Entity
      • Invite An Entity
      • Generate token
      • Generate admin token
      • Get An Entity
      • Get An Entity By Id
      • Update An Entity
      • Create A Property Of An Entity
      • Update A Property Of An Entity
      • Revoke a Credential
      • Delete An Entity
    • Schema
      • Create Schema
      • Get Schema
      • Update Schema
      • Delete Schema
      • Publish A Schema
    • Attestation API
      • Raise An Attestation
      • Get Attestation Certificate
    • Claims API
      • Get All Claims
      • Get Claim by ID
      • Attest A Claim
    • Discovery API
      • Search An Entity
    • File Storage API
      • Upload A File
      • Get Uploaded File
      • Delete A File/ Multiple Files
    • Bulk Issuance API
      • Get Sample Template
      • Upload CSV
      • Get all uploaded Files
      • Download a Report File
    • Metrics APIs
      • Get Count
      • Get Aggregates
    • Other APIs
      • Sign API
      • Verify API
      • Swagger JSON API
      • Health API
  • Reference Solutions
    • Education
      • Education Ecosystem
        • Installation
      • Education Registries
        • Installation
    • Certificate Issuance
      • Installation(WIP)
      • User Guide
    • eLocker
      • High Level Diagram
      • Installation (WIP)
        • Frontend Setup E-locker
      • User Guide
    • Health Registries
      • Organ Registries
        • Frontend Setup
        • Backend Setup
        • User Guide
    • Vaccination Platform
    • Skills & Work Credentials
    • Govt to Person (G2P)
    • Unified Learners Passport (ULP)
      • ULP Capabilities
      • Example Scenario
      • Technical Components (WIP)
      • Demo/Sandbox Links (WIP)
      • Installation Guide (WIP)
        • Frontend Setup
        • Installation through docker-compose
        • Dummy records setup for refrence
  • Links
    • Source Code
    • Releases & Changelogs
    • Website
    • Roadmap
    • Reference links
    • Design
  • Community
    • Discussion Forum
    • Contributing
    • Contributors
    • Contribution Guidebook
    • Code of Conduct
    • Community Events
    • Status By Track
  • HELP
    • Roadmap
    • FAQs
    • Glossary
Powered by GitBook

Copyright (c) 2023 EkStep Foundation under MIT License

On this page
  1. Developer Documentation
  2. Backup and Restore
  3. PostgreSQL

Continuous Archiving and Point-in-Time Recovery (PITR)

PostgreSQL maintains a write-ahead log (WAL) in the pg_wal/ subdirectory of the data directory to record all changes made to the database's data files. This log ensures crash safety and allows restoring consistency by replaying the log entries since the last checkpoint. Combining a file-system-level backup with a backup of the WAL files provides an alternative backup strategy. To recover, the file system backup is restored, and then the backed-up WAL files are replayed to bring the system to the current state. This approach offers some benefits:

  1. A perfectly consistent file system backup is not required as internal inconsistencies can be corrected through log replay.

  2. Continuous backup can be achieved by archiving the WAL files, which is useful for large databases where frequent full backups may not be convenient.

  3. Point-in-time recovery is supported, allowing the database to be restored to any desired state since the base backup.

  4. By continuously feeding the series of WAL files to another machine with the same base backup, a warm standby system can be created for nearly-current database copies.

Note that pg_dump and pg_dumpall cannot be used as part of continuous archiving since they do not produce file-system-level backups with enough information for WAL replay.

It's important to consider that this method only supports the restoration of an entire database cluster and requires significant archival storage for the base backup and WAL traffic. However, it is a preferred backup technique in situations where high reliability is crucial.

To successfully recover using continuous archiving, a continuous sequence of archived WAL files extending back to the start time of the backup is necessary. Therefore, it's essential to set up and test the procedure for archiving WAL files before taking the initial base backup.

Limitations

Currently, the continuous archiving technique in PostgreSQL has some limitations that may be addressed in future releases. These limitations include:

  1. Risk of template database modifications during base backup: If a CREATE DATABASE command is executed while a base backup is in progress and the template database is modified during this time, the modifications may be applied to the created database during recovery. To avoid this issue, it is recommended not to modify any template databases while taking a base backup.

  2. Tablespaces replaying with absolute paths: CREATE TABLESPACE commands are logged with absolute paths and will be replayed as tablespace creations with the same absolute path. This can be problematic if the log is replayed on a different machine or even on the same machine into a new data directory, as it will overwrite the contents of the original tablespace. To avoid potential issues, it is best practice to take a new base backup after creating or dropping tablespaces.

  3. Bulky default WAL format: The default Write-Ahead Log (WAL) format is relatively large because it includes disk page snapshots to support crash recovery. These page snapshots ensure the recovery process can fix partially-written disk pages. However, depending on the hardware and software setup, the risk of partial writes may be negligible. In such cases, the total volume of archived logs can be significantly reduced by disabling page snapshots using the full_page_writes parameter. It's essential to carefully read the notes and warnings in Chapter 30 before making this change. Disabling page snapshots does not prevent the use of logs for point-in-time recovery (PITR). Future development may focus on compressing archived WAL data by removing unnecessary page copies even with full_page_writes enabled. Administrators can also reduce the number of page snapshots included in WAL by increasing the checkpoint interval parameters as much as possible.

PreviousFile System Level BackupNextCassandra

Last updated 1 year ago