在上一篇文章中，介绍了如何使用python导入数据，导入数据后的第二步往往就是数据清洗，下面我们来看看如何使用pandas进行数据清洗工作

导入相关库

import pandas as pd

dataframe = pd.read_csv(r'C:/Users/DELL/data-science-learning/python数据分析笔记/探索性数据分析/train.csv')
dataframe.head(5)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th…	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S

🥇1.总览数据

• 查看数据维度

dataframe.shape

(891, 12)

• 描述性统计分析

dataframe.describe()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	PassengerId	Survived	Pclass	Age	SibSp	Parch	Fare
count	891.000000	891.000000	891.000000	714.000000	891.000000	891.000000	891.000000
mean	446.000000	0.383838	2.308642	29.699118	0.523008	0.381594	32.204208
std	257.353842	0.486592	0.836071	14.526497	1.102743	0.806057	49.693429
min	1.000000	0.000000	1.000000	0.420000	0.000000	0.000000	0.000000
25%	223.500000	0.000000	2.000000	20.125000	0.000000	0.000000	7.910400
50%	446.000000	0.000000	3.000000	28.000000	0.000000	0.000000	14.454200
75%	668.500000	1.000000	3.000000	38.000000	1.000000	0.000000	31.000000
max	891.000000	1.000000	3.000000	80.000000	8.000000	6.000000	512.329200

🥈2.筛选数据

• 过滤所有女性和年龄大于60岁的乘客

dataframe[(dataframe['Sex'] == 'female') & (dataframe['Age']>=60)]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
275	276	1	1	Andrews, Miss. Kornelia Theodosia	female	63.0	1	0	13502	77.9583	D7	S
366	367	1	1	Warren, Mrs. Frank Manley (Anna Sophia Atkinson)	female	60.0	1	0	110813	75.2500	D37	C
483	484	1	3	Turkula, Mrs. (Hedwig)	female	63.0	0	0	4134	9.5875	NaN	S
829	830	1	1	Stone, Mrs. George Nelson (Martha Evelyn)	female	62.0	0	0	113572	80.0000	B28	NaN

可以看出，一共有四名年龄大于60岁的女性乘客

🥉3.替换数据

• 将female换成woman，将male换成man

dataframe['Sex'].replace(['female','male'],['woman','man']).head(5)

0      man
1    woman
2    woman
3    woman
4      man
Name: Sex, dtype: object

🏅4.更改列名

• 查看所有列名

dataframe.columns

Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')

• 重命名列

	PassengerId	Survived	Passenger Class	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th…	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S

dataframe.rename(columns={'Pclass':'Passenger Class','Sex':'Gender'}).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	PassengerId	Survived	Passenger Class	Name	Gender	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th…	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S

🥇5.查找唯一值

在pandas中，我们可以使用unique()查找唯一值

# 查找唯一值
dataframe['Sex'].unique()

array(['male', 'female'], dtype=object)

# 显示唯一值出现的个数
dataframe['Sex'].value_counts()

male      577
female    314
Name: Sex, dtype: int64

# 查找类型票的数量
dataframe['Pclass'].value_counts()

3    491
1    216
2    184
Name: Pclass, dtype: int64

# 查找唯一值的种类
dataframe['Pclass'].nunique()

3

🥈6.查找缺失值

# 查找空数据
dataframe[dataframe['Age'].isnull()].head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
5	6	0	3	Moran, Mr. James	male	NaN	0	0	330877	8.4583	NaN	Q
17	18	1	2	Williams, Mr. Charles Eugene	male	NaN	0	0	244373	13.0000	NaN	S
19	20	1	3	Masselmani, Mrs. Fatima	female	NaN	0	0	2649	7.2250	NaN	C
26	27	0	3	Emir, Mr. Farred Chehab	male	NaN	0	0	2631	7.2250	NaN	C
28	29	1	3	O’Dwyer, Miss. Ellen “Nellie”	female	NaN	0	0	330959	7.8792	NaN	Q

pandas没有NaN 如果想要处理的话必须导入numpy包

import numpy as np
dataframe['Sex'].replace('male',np.nan).head()

0       NaN
1    female
2    female
3    female
4       NaN
Name: Sex, dtype: object

🥉7.删除列或行

# 删除一列，采用drop方法，并传入参数axis
dataframe.drop('Age',axis=1).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	PassengerId	Survived	Pclass	Name	Sex	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th…	female	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	0	0	373450	8.0500	NaN	S

#删除行
dataframe.drop(1)

# 删除重复行 使用subset参数指明要删除的列
dataframe.drop_duplicates(subset='Sex').head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Name	PClass	Age	Sex	Survived	SexCode
0	Allen, Miss Elisabeth Walton	1st	29.0	female	1	1
2	Allison, Mr Hudson Joshua Creighton	1st	30.0	male	0	0

🏅8. groupby分组

• 计算男性和女性的平均值

==思路一==，将所有男性和女性的条件进行选取分别计算

man = dataframe[dataframe['Sex']=='male']
woman = dataframe[dataframe['Sex']=='female']
print(man.mean())
print(woman.mean())

Age         31.014338
Survived     0.166863
SexCode      0.000000
dtype: float64
Age         29.396424
Survived     0.666667
SexCode      1.000000
dtype: float64

==思路二==，用groupby方法简化

dataframe.groupby('Sex').mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Age	Survived	SexCode
Sex
—	—	—	—
female	29.396424	0.666667	1.0
male	31.014338	0.166863	0.0

# 按行分组，计算行数
dataframe.groupby('Sex')['Name'].count()

Sex
female    462
male      851
Name: Name, dtype: int64

dataframe.groupby(['Sex','Survived']).mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

		PassengerId	Pclass	Age	SibSp	Parch	Fare
Sex	Survived
—	—	—	—	—	—	—	—
female	0	434.851852	2.851852	25.046875	1.209877	1.037037	23.024385
1	429.699571	1.918455	28.847716	0.515021	0.515021	51.938573
male	0	449.121795	2.476496	31.618056	0.440171	0.207265	21.960993
1	475.724771	2.018349	27.276022	0.385321	0.357798	40.821484

🥇9.按照时间段来进行分组

• 使用resample参数来进行取样本

# 创建时期范围
time_index = pd.date_range('06/06/2017', periods=100000, freq='30S') # periods表示有多少数据，freq表示步长

dataframe = pd.DataFrame(index=time_index)

# 创建一个随机变量
dataframe['Sale_Amout'] = np.random.randint(1, 10, 100000)

# resample 参数，按周对行分组，计算每一周的总和
dataframe.resample('W').sum()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Sale_Amout
2017-06-11	86292
2017-06-18	100359
2017-06-25	100907
2017-07-02	100868
2017-07-09	100522
2017-07-16	10478

# 使用resample可以按一组时间间隔来进行分组，然后计算每一个时间组的某个统计量
dataframe.resample('2W').mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Sale_Amout
2017-06-11	4.993750
2017-06-25	4.991716
2017-07-09	4.994792
2017-07-23	5.037500

dataframe.resample('M').count()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Sale_Amout
2017-06-30	72000
2017-07-31	28000

# resample默认是以最后一个数据作 使用label参数可以进行调整
dataframe.resample('M', label='left').count()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Sale_Amout
2017-05-31	72000
2017-06-30	28000

🥈10.遍历一个列的数据

dataframe = pd.read_csv(url)

# 以大写的形势打印前两行的名字
for name in dataframe['Name'][0:2]:
    print(name.upper())

ALLEN, MISS ELISABETH WALTON
ALLISON, MISS HELEN LORAINE

🥉11.对一列的所有元素应用某个函数

def uppercase(x):
    return x.upper()

dataframe['Name'].apply(uppercase)[0:2]

0    ALLEN, MISS ELISABETH WALTON
1     ALLISON, MISS HELEN LORAINE
Name: Name, dtype: object

🏅12. pandas高级函数

dataframe.groupby('Sex').apply(lambda x:x.count())

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Name	PClass	Age	Sex	Survived	SexCode
Sex
—	—	—	—	—	—	—
female	462	462	288	462	462	462
male	851	851	468	851	851	851

通过联合使用groupby 和apply，我们就能计算自定义的统计量 例如上面我们发现age、cabin具有大量的缺失值

🥇13. 连接多个Dataframe

data_a = {'id':['1', '2', '3'],
          'first': ['Alex', 'Amy', 'Allen'],
          'last': ['Anderson', 'Ackerman', 'Ali']}
dataframe_a = pd.DataFrame(data_a, columns=['id','first', 'last'])

data_b = {'id':['4', '5', '6'],
          'first': ['Billy', 'Brian', 'Bran'],
          'last': ['Bonder', 'Black', 'Balwner']}
dataframe_b = pd.DataFrame(data_b, columns=['id','first', 'last'])

pd.concat([dataframe_a, dataframe_b], axis=0)#在行的方向进行

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	id	first	last
0	1	Alex	Anderson
1	2	Amy	Ackerman
2	3	Allen	Ali
0	4	Billy	Bonder
1	5	Brian	Black
2	6	Bran	Balwner

pd.concat([dataframe_a, dataframe_b], axis=1)#在列的方向进行

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	id	first	last	id	first	last
0	1	Alex	Anderson	4	Billy	Bonder
1	2	Amy	Ackerman	5	Brian	Black
2	3	Allen	Ali	6	Bran	Balwner

# 也可以用append方法进行添加
c = pd.Series([10, 'Chris', 'Chillon'], index=['id','first','last'])

dataframe.append(c, ignore_index=True)#如果c原来有名字忽略

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	id	first	last
0	1	Alex	Anderson
1	2	Amy	Ackerman
2	3	Allen	Ali
3	10	Chris	Chillon

如果你对Python感兴趣，想要学习python，这里给大家分享一份Python全套学习资料，都是我自己学习时整理的，希望可以帮到你，一起加油！

😝有需要的小伙伴，可以点击下方链接免费领取或者V扫描下方二维码免费领取🆓
Python全套学习资料

1️⃣零基础入门

① 学习路线

对于从来没有接触过Python的同学，我们帮你准备了详细的学习成长路线图。可以说是最科学最系统的学习路线，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。

② 路线对应学习视频

还有很多适合0基础入门的学习视频，有了这些视频，轻轻松松上手Python~

③练习题

每节视频课后，都有对应的练习题哦，可以检验学习成果哈哈！

2️⃣国内外Python书籍、文档

① 文档和书籍资料

3️⃣Python工具包+项目源码合集

①Python工具包

学习Python常用的开发软件都在这里了！每个都有详细的安装教程，保证你可以安装成功哦！

②Python实战案例

光学理论是没用的，要学会跟着一起敲代码，动手实操，才能将自己的所学运用到实际当中去，这时候可以搞点实战案例来学习。100+实战案例源码等你来拿！

③Python小游戏源码

如果觉得上面的实战案例有点枯燥，可以试试自己用Python编写小游戏，让你的学习过程中增添一点趣味！

4️⃣Python面试题

我们学会了Python之后，有了技能就可以出去找工作啦！下面这些面试题是都来自阿里、腾讯、字节等一线互联网大厂，并且有阿里大佬给出了权威的解答，刷完这一套面试资料相信大家都能找到满意的工作。

5️⃣Python兼职渠道

而且学会Python以后，还可以在各大兼职平台接单赚钱，各种兼职渠道+兼职注意事项+如何和客户沟通，我都整理成文档了。

上述所有资料 ⚡️ ，朋友们如果有需要的，可以扫描下方👇👇👇二维码免费领取🆓