地震部门数字档案馆系统搭建与数据迁移实操指南

系统架构设计与技术选型

地震部门数字档案馆需要处理海量的地震监测数据、报告文档和历史档案，核心需求包括高可靠性存储、快速检索和长期保存。我们采用微服务架构，将系统拆分为独立可扩展的模块。

基础环境配置

操作系统选择Ubuntu Server 22.04 LTS，确保长期稳定支持。使用以下命令安装基础依赖：

``` sudo apt update sudo apt install -y docker.io docker-compose nginx postgresql-client redis-tools sudo systemctl enable docker ```

核心组件选型

• 文件存储：MinIO对象存储，替代传统文件系统，支持S3协议
• 数据库：PostgreSQL 15 + TimescaleDB扩展，用于时序数据和元数据存储
• 全文检索：Elasticsearch 8.x，支持地震报告全文检索
• 缓存层：Redis 7.x，提升系统响应速度
• 后端框架：Spring Boot 3.1 + Java 17
• 前端框架：Vue 3 + Element Plus

详细部署步骤

1. 存储层部署

创建docker-compose-storage.yml文件：

``` version: '3.8' services: minio: image: minio/minio:latest command: server /data --console-address ":9001" ports: - "9000:9000" - "9001:9001" environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: YourStrongPassword123! volumes: - minio_data:/data restart: always postgres: image: timescale/timescaledb:latest-pg15 environment: POSTGRES_DB: seismic_archive POSTGRES_USER: archive_admin POSTGRES_PASSWORD: DbPassword456! ports: - "5432:5432" volumes: - pg_data:/var/lib/postgresql/data restart: always elasticsearch: image: elasticsearch:8.11.0 environment: - discovery.type=single-node - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ports: - "9200:9200" volumes: - es_data:/usr/share/elasticsearch/data restart: always volumes: minio_data: pg_data: es_data: ```

启动存储服务：docker-compose -f docker-compose-storage.yml up -d

2. 数据库初始化

创建数据库表结构，保存为init_schema.sql：

``` -- 创建地震数据表 CREATE TABLE seismic_events ( event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), event_time TIMESTAMPTZ NOT NULL, latitude DECIMAL(9,6) NOT NULL, longitude DECIMAL(9,6) NOT NULL, depth DECIMAL(6,2), magnitude DECIMAL(3,1), location_description TEXT, created_at TIMESTAMPTZ DEFAULT NOW() ); -- 启用TimescaleDB扩展 CREATE EXTENSION IF NOT EXISTS timescaledb; -- 将地震事件表转换为超表 SELECT create_hypertable('seismic_events', 'event_time'); -- 创建档案元数据表 CREATE TABLE archive_metadata ( file_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), original_filename VARCHAR(500) NOT NULL, file_size BIGINT NOT NULL, mime_type VARCHAR(100), storage_path VARCHAR(1000) NOT NULL, seismic_event_id UUID REFERENCES seismic_events(event_id), upload_time TIMESTAMPTZ DEFAULT NOW(), metadata JSONB DEFAULT '{}'::jsonb ); -- 创建索引 CREATE INDEX idx_seismic_time ON seismic_events(event_time); CREATE INDEX idx_seismic_location ON seismic_events USING gist(ll_to_earth(latitude, longitude)); CREATE INDEX idx_metadata_event ON archive_metadata(seismic_event_id); CREATE INDEX idx_metadata_json ON archive_metadata USING gin(metadata); ```

执行初始化：psql -h localhost -U archive_admin -d seismic_archive -f init_schema.sql

3. 后端服务部署

创建Spring Boot应用的application.yml配置文件：

``` server: port: 8080 spring: datasource: url: jdbc:postgresql://localhost:5432/seismic_archive username: archive_admin password: DbPassword456! driver-class-name: org.postgresql.Driver jpa: hibernate: ddl-auto: validate properties: hibernate: dialect: org.hibernate.dialect.PostgreSQLDialect jdbc: batch_size: 20 format_sql: true minio: endpoint: http://localhost:9000 accessKey: admin secretKey: YourStrongPassword123! bucket: seismic-archive elasticsearch: uris: http://localhost:9200 file: upload: max-size: 2GB allowed-types: application/pdf,image/tiff,image/jpeg,application/msword,application/vnd.openxmlformats-officedocument.wordprocessingml.document ```

核心文件上传控制器代码：

``` @RestController @RequestMapping("/api/archive") public class ArchiveController { @PostMapping("/upload") public ResponseEntity uploadFile( @RequestParam("file") MultipartFile file, @RequestParam("eventId") String eventId, @RequestParam(value = "metadata", required = false) String metadataJson) { // 验证文件类型 if (!isValidFileType(file.getContentType())) { return ResponseEntity.badRequest().build(); } // 生成存储路径 String objectName = generateStoragePath(eventId, file.getOriginalFilename()); // 上传到MinIO minioClient.putObject( PutObjectArgs.builder() .bucket("seismic-archive") .object(objectName) .stream(file.getInputStream(), file.getSize(), -1) .contentType(file.getContentType()) .build() ); // 保存元数据到PostgreSQL ArchiveMetadata metadata = new ArchiveMetadata(); metadata.setOriginalFilename(file.getOriginalFilename()); metadata.setFileSize(file.getSize()); metadata.setStoragePath(objectName); metadata.setSeismicEventId(UUID.fromString(eventId)); if (metadataJson != null) { metadata.setMetadata(objectMapper.readValue(metadataJson, JsonNode.class)); } metadataRepository.save(metadata); // 索引到Elasticsearch indexDocument(metadata); return ResponseEntity.ok(new ArchiveResponse(metadata.getFileId())); } private String generateStoragePath(String eventId, String filename) { LocalDate today = LocalDate.now(); return String.format("%s/%s/%s/%s", today.getYear(), today.getMonthValue(), eventId, UUID.randomUUID() + getExtension(filename) ); } } ```

4. 前端界面配置

创建Vue组件用于文件上传：

``` ```

数据迁移与导入流程

1. 历史数据迁移脚本

创建Python迁移脚本migrate_legacy_data.py：

``` import os import psycopg2 from minio import Minio from minio.error import S3Error import json from pathlib import Path def migrate_file_system_to_minio(local_path, minio_client, bucket_name): """迁移本地文件到MinIO对象存储""" for root, dirs, files in os.walk(local_path): for file in files: file_path = os.path.join(root, file) relative_path = os.path.relpath(file_path, local_path) 获取文件元数据 file_stat = os.stat(file_path) 上传到MinIO try: minio_client.fput_object( bucket_name, f"legacy/{relative_path}", file_path, metadata={ 'original-filename': file, 'file-size': str(file_stat.st_size), 'migration-date': '2024-01-15' } ) print(f"上传成功: {relative_path}") 记录到数据库 record_metadata_in_db(relative_path, file, file_stat.st_size) except S3Error as e: print(f"上传失败 {relative_path}: {e}") def record_metadata_in_db(storage_path, filename, file_size): """记录文件元数据到PostgreSQL""" conn = psycopg2.connect( host="localhost", database="seismic_archive", user="archive_admin", password="DbPassword456!" ) cursor = conn.cursor() cursor.execute(""" INSERT INTO archive_metadata (original_filename, file_size, storage_path, upload_time) VALUES (%s, %s, %s, NOW()) """, (filename, file_size, storage_path)) conn.commit() cursor.close() conn.close() if __name__ == "__main__": 初始化MinIO客户端 minio_client = Minio( "localhost:9000", access_key="admin", secret_key="YourStrongPassword123!", secure=False ) 确保存储桶存在 if not minio_client.bucket_exists("seismic-archive"): minio_client.make_bucket("seismic-archive") 迁移数据 migrate_file_system_to_minio( "/path/to/legacy/files", minio_client, "seismic-archive" ) ```

2. 批量导入地震事件数据

创建CSV导入脚本import_seismic_events.py：

``` import pandas as pd from sqlalchemy import create_engine from datetime import datetime def import_seismic_events_from_csv(csv_file): """从CSV文件导入地震事件数据""" 读取CSV文件 df = pd.read_csv(csv_file, parse_dates=['event_time']) 数据清洗和转换 df['created_at'] = datetime.now() 使用SQLAlchemy批量插入 engine = create_engine( 'postgresql://archive_admin:DbPassword456!@localhost:5432/seismic_archive' ) 批量插入数据 df.to_sql('seismic_events', engine, if_exists='append', index=False) print(f"成功导入 {len(df)} 条地震事件记录") 执行导入 import_seismic_events_from_csv('seismic_events_2023.csv') ```

系统维护与监控

1. 备份策略配置

创建备份脚本backup_archive.sh：

``` !/bin/bash BACKUP_DIR="/backup/seismic-archive" DATE=$(date +%Y%m%d_%H%M%S) 备份PostgreSQL数据库 pg_dump -h localhost -U archive_admin seismic_archive > \ $BACKUP_DIR/db_backup_$DATE.sql 压缩数据库备份 gzip $BACKUP_DIR/db_backup_$DATE.sql 备份Elasticsearch索引 curl -X POST "localhost:9200/_snapshot/archive_backup/snapshot_$DATE?wait_for_completion=true" 保留最近30天的备份 find $BACKUP_DIR -name ".sql.gz" -mtime +30 -delete echo "备份完成: $DATE" ```

设置定时任务：crontab -e添加：

``` 0 2 /usr/local/bin/backup_archive.sh >> /var/log/archive_backup.log 2>&1 ```

2. 监控配置

创建Prometheus监控配置prometheus.yml：

``` global: scrape_interval: 15s scrape_configs: - job_name: 'postgres' static_configs: - targets: ['localhost:9187'] - job_name: 'minio' static_configs: - targets: ['localhost:9000'] - job_name: 'elasticsearch' static_configs: - targets: ['localhost:9200'] - job_name: 'application' static_configs: - targets: ['localhost:8080'] ```

故障排查与优化

常见问题解决

• MinIO上传失败：检查S3端点配置和访问密钥，确保存储桶存在
• 数据库连接超时：检查PostgreSQL服务状态和防火墙设置
• 文件上传大小限制：调整Spring Boot的max-file-size和max-request-size配置
• 检索性能问题：为Elasticsearch添加更多索引，优化查询语句

性能优化建议

• 为频繁查询的地震事件字段创建复合索引
• 使用MinIO的分层存储策略，将冷数据转移到低成本存储
• 配置Elasticsearch分片和副本，根据数据量调整集群规模
• 实现CDN加速，为频繁访问的档案文件提供快速下载