分布式文件系统之FastDFS

3年前 (2017-10-02) gtj linux, WEB应用, 分布式存储 0评论 已收录 268℃

一.简介:

fastDFS c语言编写的轻量级分布式文件系统。作者余庆(happyfish100),支持 Linux、FreeBSD、AID 等 Unix 系统。GitHub地址:https://github.com/happyfish100/
FastDFS是一个开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,FastDFS为互联网量身定制,充分考虑了冗余备份、负载均衡、线性扩容等机制,并注重高可用、高性能等指标。特别适合以文件为载体的在线服务,如相册网站、视频网站等等,使用FastDFS很容易搭建一套高性能的文件服务器集群提供文件上传、下载等服务。
适用文件文件大小为-4kb 到 500MB 中小型文件系统。

二.架构图



FastDFS 系统有三个角色:跟踪服务器(Tracker Server)、存储服务器(Storage Server)和客户端(Client)。
Tracker Server: 跟踪服务器,主要做调度工作,起到均衡的作用;负责管理所有的storage server
和 group,每个storage在启动后会连接Tracker,告知自己所属group等信息,并保持周期性心跳。
Storage Server:存储服务器,主要提供容量和备份服务;以group为单位,每个group内可以有多台storage server,数据互为备份。
Client:客户端,上传下载数据的服务器,也就是我们自己的项目所部署在的服务器。

三.搭建fastdfs

1.ip规划

10.20.23.134
10.20.23.135

2.安装fastdfs

先安装依赖libfastcommon:

cd /usr/local/src
git clone https://github.com/happyfish100/libfastcommon.git
cd libfastcommon/
./mask.sh
./make.sh install
cd ..

安装fastdfs:

wget https://github.com/happyfish100/fastdfs/archive/V5.08.tar.gz
tar  xf  V5.08.tar.gz 
cd fastdfs-5.08/
./make.sh 
./make.sh install

3.配置fastdfs

cp  /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
vi /etc/fdfs/tracker.conf #追踪服务
base_path=/data/fdfs_tracker #存储路径
max_connections=256  最大链接数
store_group=group2
reserved_storage_space = 10% 存储数据百分比 
 
mkdir -p /data/fdfs_tracker
mkdir -p /data/fdfs_storage
mkdir -p /data/fdfs_storage/base
mkdir -p /data/fdfs_storage/store
/etc/init.d/fdfs_trackerd start   
 备注: 两台都要安装追踪服务
存储:
cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
vim /etc/fdfs/storage.conf
port=23000
heart_beat_interval=30 30s发一次心跳
base_path=/data/fdfs_storage/base
store_path_count=1 
store_path0=/data/fdfs_storage/store
tracker_server=10.20.23.134:22122
tracker_server=10.20.23.135:22122
scp -p 22 /etc/fdfs/storage.conf root@10.20.23.135:/etc/fdfs/
启动:
/etc/init.d/fdfs_storaged start

4.部署客户端:

cp /etc/fdfs/client.conf.sample  /etc/fdfs/client.conf
vim /etc/fdfs/client.conf
base_path=/tmp 日志路径
tracker_server=10.20.23.134:22122
tracker_server=10.20.23.135:22122

5.fastdfs基本操作

上传文件:

 fdfs_upload_file  /etc/fdfs/client.conf  /etc/passwd(上传的文件)
[root@cacti ~]# fdfs_upload_file  /etc/fdfs/client.conf  /etc/passwd
group1/M00/00/00/ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635
路径:/data/fdfs_storage/store/data

md5sum ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635

下载这个文件

fdfs_download_file /etc/fdfs/client.conf  group1/M00/00/00/ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635
[root@cacti ~]# md5sum  ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635 
e47a685330ed36c73918d8084571911e  ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635
[root@cacti ~]# md5sum  /etc/pa
pam.d/   pango/   passwd   passwd-  
[root@cacti ~]# md5sum  /etc/passwd
e47a685330ed36c73918d8084571911e  /etc/passwd

查看详细内容:

fdfs_file_info /etc/fdfs/client.conf  group1/M00/00/00/ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635

删除文件:

fdfs_delete_file /etc/fdfs/client.conf  group1/M00/00/00/ChQXhlhv-SGAHEm_AAAEjqtd9Vw1021635

文件的追加:

fdfs_upload_appender
例子:
 echo "hello ">append.txt
 echo "world">append2.txt
fdfs_upload_appender /etc/fdfs/client.conf append.txt 
group1/M00/00/00/ChQXh1hy8ZOEIHjmAAAAAHcc3SA802.txt
fdfs_append_file  /etc/fdfs/client.conf   group1/M00/00/00/ChQXhlhv_gqEZJtiAAAAAHcc3SA896.txt   append.txt 
fdfs_append_file  /etc/fdfs/client.conf   group1/M00/00/00/ChQXhlhv_gqEZJtiAAAAAHcc3SA896.txt   append2.txt 
fdfs_download_file  /etc/fdfs/client.conf   group1/M00/00/00/ChQXhlhv_gqEZJtiAAAAAHcc3SA896.txt
 cat ChQXhlhv_gqEZJtiAAAAAHcc3SA896.txt
hello
hello
world 

文件罗列

 fdfs_monitor:
 罗列详细信息:可以通过这个命令做监控. 
 fdfs_monitor /etc/fdfs/client.conf

节点down的情况下可以踢掉节点

 fdfs_monitor /etc/fdfs/client.conf  delete group1 10.20.23.134

6.结合php

 cd /usr/local/src/fastdfs-5.08/php_client/
/usr/bin/phpize
./configure --with-php-config=/usr/bin/php-config 
 make  && make install 
 
 ll  /usr/local/src/fastdfs-5.08/php_client/modules
total 328
-rwxr-xr-x. 1 root root 333944 Jan  9 11:07 fastdfs_client.so

cat  fastdfs_client.ini >>/etc/php.ini 
测试php客户端是否正常:
 /usr/bin/php fastdfs_test.php 

7.安装java的客户端:

 yum install ant
 git clone https://github.com/happyfish100/fastdfs-client-java.git
 https://github.com/happyfish100/fastdfs-client-java/tree/master/src
 
 fastdfs-nginx-module:
 git clone https://github.com/happyfish100/fastdfs-nginx-module.git
 ./configure --prefix=/usr/local/nginx  --user=www  --group=www --with-http_ssl_module  --with-pcre=/usr/local/src/pcre-8.31 --add-module=/usr/local//src/fastdfs-nginx-module/src/
 make  &&make install

8.通过nginx直接查看fastdfs内容

cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
cd /usr/local/src/fastdfs-5.08/conf/
cp  anti-steal.jpg  http.conf   mime.types  /etc/fdfs/
touch /var/log/mod_fastdfs.log
chown  www:www  /var/log/mod_fastdfs.log 
vi /usr/local/nginx/conf/nginx.conf
 server {
        listen       80;
        server_name  10.20.23.135;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location /groups/M00 {
            root   /data/fdfs_storage/store;
            ngx_fastdfs_module;
            
        }

 /usr/local/nginx/sbin/nginx

9.测试

可以网页访问
http://10.20.23.135/group1/M00/00/00/ChQXh1hzOrqAQeZ2AAKIE_kf4YA453.jpg
能够查看到图片

博主

这货来去如风,什么鬼都没留下!!!

相关推荐

嗨、骚年、快来消灭0回复。

×
订阅图标按钮
Less is more!!!